⚡ TL;DR — Key Takeaways
- The Problem: AI bots scrape content to train models, stealing traffic and reducing server performance.
- The Fix: Use
robots.txtin Rank Math and cloud firewall settings in Cloudflare to block Claude AI and OpenAI bots’ access.
- The Result: Your site remains indexed by search engines while your content is protected from unauthorized AI scraping.
Table of Contents
AI crawlers are scraping the internet daily to train large language models. While some crawlers help index your site for search, others take your content without offering any citation or search traffic in return. Here we will discuss about how to block Claude AI and OpenAI bots from scraping valuable information from your websites.
Even after posting high-quality content on my blog, I noticed my traffic stagnated for a long time. When I looked deeper into my data, I discovered a massive spike in bot scraping logs. That is when I realized AI bots were scraping my website and using my content to answer user queries directly—giving me absolutely zero traffic or credit. That is why it is absolutely essential to block Claude AI and OpenAI bots from scraping your website for free.
If you want to protect your copyrighted articles from unauthorized AI training models, you can stop them using two highly effective methods: robots.txt and network firewalls. Here is exactly how to do it in under 5 minutes.
Method 1: Update Your WordPress Robots.txt File
The fastest way to ask AI crawlers to leave your site alone is by modifying your robots.txt file. This file sits at the root of your website and contains instructions for automated web bots.
If you use Rank Math SEO, you do not need to access your server files directly. Simply log into your WordPress Dashboard, go to Rank Math > General Settings > Edit Robots.txt, and paste the following rules:
Text
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Use code with cautionRefer this for more such rules: robots.txt setting
Why these specific bot rules matter
- GPTBot & ChatGPT-User: These are OpenAI’s primary web crawlers used to gather data for ChatGPT models.
- ClaudeBot & Claude-SearchBot: These belong to Anthropic and crawl sites specifically for Claude training data.
- Google-Extended: This rule allows you to block Google’s Gemini training systems without hurting your standard Google Search visibility.
To block Claude AI and OpenAI bots, the rule above is sufficient. But other malicious scrapers don’t respect the robots.txt rules. They still scrape your website. To put a blanket stop to malicious AI bots, the below given method is beneficial.
Method 2: Block AI Bots at the Network Level (Cloudflare)
To completely block stubborn AI bots that bypass your rules, you need a firewall. If your WordPress site uses Cloudflare, you can activate network-level protection with one click.
Cloudflare recently updated its bot management capabilities. To protect your site, navigate to your Cloudflare dashboard and complete these steps:
- Select your website domain.
- Navigate to Security > Bots in the left sidebar.
- Locate the toggle switch for Block AI bots.
- Toggle the setting to On.
This system blocks verified AI training bots instantly. Additionally, starting September 15, 2026, Cloudflare will automatically block training and agent bots by default on ad-supported pages. This provides automated, long-term security for content creators.
Conclusion
Combining a clear robots.txt directive with a Cloudflare firewall gives your site a multi-layered defense against data scraping. Your content remains indexed for human searchers, but safe from AI model training loops.
Take a close look at your website analytics today—are you seeing unexpected spikes in your traffic logs? Drop a comment below and let me know which AI crawlers are hitting your site the hardest, or if you ran into any issues setting up your Cloudflare firewall and block Claude AI and OpenAI bots.
Related: AI Security in 2026: 20 Numbers That Show Who’s Actually Winning – Enterprises are pouring $49 billion into AI-powered defense but only $2.8 billion into securing the AI itself — a 17-to-1 bet that’s about to get tested.
DHS HSIN Breach: Hackers Sat Inside World Cup Security for 5 Weeks – Hackers breached DHS’s Homeland Security Information Network at the worst possible time, exposing how one overlooked system can put major events like the FIFA World Cup at greater cyber risk.
Claude AI Went From $1B to $30B in Two Years. Its Security Story Is Just as Wild. – Claude went from $1 billion to $30 billion in two years by being brilliant at finding security flaws — the same brilliance a nation-state group used to jailbreak it for a live espionage campaign.
Frequently Asked Questions (FAQ)
Q1. Will blocking AI bots hurt my website rankings on Google Search?
No, the Google-Extended rule specifically targets AI training without hindering regular Googlebot indexing.
Q2. What happens if an AI bot ignores my robots.txt file?
While robots.txt acts as a request, using Method 2 (Cloudflare) blocks malicious scrapers at the firewall level.
Q3. Do I need to update this list when new AI bots come out?
Yes, updating the bot list every few months ensures protection against new AI crawlers.
Q4. Is it completely free to block AI bots using these methods?
Yes, both Rank Math robots.txt edits and Cloudflare’s AI bot blocking toggle are free.
Q5. What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s automated crawler that crawls the web to gather data for training future AI models. ChatGPT-User is used when a ChatGPT user explicitly asks the AI to visit a specific link or fetch live web information for them.
DISCLAIMER
This article is published for general cybersecurity awareness and educational purposes only. The information contained herein is based on publicly available threat intelligence research and media reporting as of July 2026. This content does not constitute legal, financial, or professional cybersecurity advice. Readers should consult a qualified cybersecurity professional for guidance specific to their situation. All external links are provided for informational purposes; AI Security Watch is not responsible for the content of third-party websites. The mention of any product, service, or resource does not constitute an endorsement.Disclaimer