The short version: you can block most AI bots with a few lines in robots.txt and a firewall rule at your CDN, but no method is bulletproof because robots.txt is a request, not a lock, and some bots simply ignore it. Do it anyway, because it stops the well-behaved majority (GPTBot, ClaudeBot, Google-Extended) and it puts you on record if you ever need to prove non-consent. Just know that blocking everything can also make you invisible in AI search answers, and that trade-off is yours to make with open eyes, not something to discover by accident.
What’s hoovering up your content right now
Most people picture “AI scraping” as one shadowy thing. It isn’t. It’s a named list of specific bots, each with a specific job:
- GPTBot – OpenAI’s crawler, used to gather training data for future models.
- ClaudeBot and anthropic-ai – Anthropic’s crawlers, same purpose.
- Google-Extended – controls whether Google can use your content for Gemini and AI Overviews, separate from normal Googlebot indexing.
- CCBot – runs Common Crawl, a huge open dataset that most AI labs, including smaller ones you’ve never heard of, train on.
- Bytespider – ByteDance’s crawler, notorious for ignoring crawl-delay settings and hitting sites hard.
- PerplexityBot and Amazonbot – used for live retrieval and answer generation, not just training.
Each of these identifies itself with a user-agent string, which means each one can be named and blocked individually. That’s the good news. The bad news is there are new ones appearing every few months, so a block list written in early 2025 is already out of date by 2026.
Step one: fix your robots.txt (five minutes, real code)
This is the baseline every guide tells you to do, so I’ll be specific rather than vague. Open or create the robots.txt file at the root of your domain (yoursite.com/robots.txt) and add:
- User-agent: GPTBot / Disallow: /
- User-agent: ClaudeBot / Disallow: /
- User-agent: Google-Extended / Disallow: /
- User-agent: CCBot / Disallow: /
- User-agent: Bytespider / Disallow: /
- User-agent: PerplexityBot / Disallow: /
- User-agent: Amazonbot / Disallow: /
Each pair goes on its own line, no blank line between the two lines of a pair. You can find the current, maintained list of AI crawler user-agents through Dark Visitors or the Cloudflare radar bot page, both of which update faster than any blog post can. Set a calendar reminder to check that list every quarter, because a new bot launching and you not knowing its name is the single most common reason this whole exercise fails quietly.
The catch, and I mean this plainly: robots.txt works only on bots that choose to read and obey it. It’s a courtesy sign, not a locked gate. Reputable labs mostly respect it because getting caught ignoring it is a PR problem for them. Smaller, scrappier scraping operations, the ones feeding cheap AI tools nobody’s heard of, often don’t bother checking it at all.
Step two: block at the server or CDN, not just the front door
If you want actual enforcement rather than a polite request, you need to block at a level the bot can’t simply skip past. Three practical options:
- Cloudflare’s AI bot block: if your site sits behind Cloudflare, even the free plan, there’s a one-click “Block AI Bots” toggle under Security settings that blocks known AI crawlers at the network edge, before they ever reach your server. This is the single highest-impact five minutes you can spend on this problem.
- .htaccess rules (Apache servers): add a RewriteCond block matching bad user-agent strings and return a 403. This works even if the bot ignores robots.txt, because you’re rejecting the request outright rather than asking nicely.
- Rate limiting by IP range: AI crawlers often hit from identifiable data centre IP ranges (AWS, Google Cloud, specific ASNs). A rate limit that caps requests per minute from a single IP catches scrapers that spoof their user-agent to look human.
If you don’t run your own server rules and this sounds like more than you want to touch yourself, this is the kind of task that’s worth twenty minutes with someone technical, and it’s one of the smaller, cheaper jobs an AI consultant for small business will set up for you alongside bigger AI strategy work, rather than a project on its own.
What happened when I checked my own logs
I’ll tell you what pushed me to sort this out, because it wasn’t a headline, it was my own server logs. I pulled a month of access logs for lilachbullock.com in late 2025 out of curiosity, and Bytespider alone had made just under 38,000 requests in that period. Zero referral traffic came back from it. Not one click. Compare that to Googlebot, which crawled a similar volume but drove actual visitors through search results. One was a trade, the other was pure extraction with nothing returned.
That’s the moment the abstract “AI is scraping everyone” story became a real number on my own site, and it’s the reason I now check crawler activity the same way I check traffic sources, monthly, not once and forgotten.
Does a DMCA notice or your terms of service stop anything?
Here’s where I’ll say something most posts on this topic won’t: legal tools are weaker than they sound. A DMCA takedown works when you can identify who’s hosting the infringing copy and where, which is straightforward for a competitor’s website republishing your blog post word for word, and nearly impossible when your paragraph has been blended into a language model’s training data and reappears, reworded, inside an AI-generated answer months later. You cannot DMCA a probability distribution.
Adding a clause to your terms of service that says “content may not be used to train AI models” gives you a stronger footing if you ever end up in a dispute, and it costs nothing to add, but it’s a paper shield, not a working fence. It matters for the record, not for prevention.
This is exactly where RSL licensing for AI content is starting to matter for small publishers. Really Simple Licensing lets you attach machine-readable licensing terms, including a price, directly to your content, so an AI company crawling your site has to see and technically respect stated terms rather than a vague blog disclaimer. It’s early, adoption is patchy, and it won’t stop a bad actor, but it’s a more concrete version of the same idea as robots.txt, this time with an actual price tag attached rather than just a request.
The trade-off nobody puts in the headline
Here’s the uncomfortable part. If you block Google-Extended, PerplexityBot, and every other AI crawler completely, you also make it much harder for your content to ever be cited or quoted inside ChatGPT, Perplexity, or Google’s AI Overviews. Those tools can’t pull an answer from a page they’re not allowed to read.
For a lot of businesses right now, being cited by name inside an AI answer is turning into a real referral channel, not unlike being quoted in a newspaper used to be. Block everything and you protect your content perfectly and get referenced by absolutely none of it.
My own answer to this, on my own site, is a middle position. I block CCBot and Bytespider outright because they’ve never sent me anything back. I leave Google-Extended and PerplexityBot unblocked on my published articles because I want to be findable inside AI answers, and I treat gated or paid material, anything behind an opt-in or a paywall, as fully off-limits to every crawler with no exceptions. That’s not a universal rule, it’s just the trade I’ve decided I’m comfortable with, and I’d rather you make that decision on purpose than have it made for you by a default setting you never looked at.
Checking who’s hitting your site
You can’t manage what you can’t see. Before you decide which bots to block, spend an hour looking at who’s visiting. Your hosting dashboard or Cloudflare analytics will show crawler activity by user-agent if you know where to look, and there’s a wider habit worth building here of checking tools most site owners never open. I keep a running list of lesser-known but useful sites, including a few log and traffic analysis tools, in my roundup of 50 websites you probably didn’t know existed, and several of them are exactly the kind of thing that turns “I think bots are scraping me” into an actual number, the same way checking my own logs turned a vague worry into a concrete 38,000.
If you want to go a step further and see how competitors are handling this, or whether they’re even bothered, the same investigative habit applies. My guide to the legal ways to check competitors’ websites covers tools that show you crawl and traffic patterns on other domains, which is a useful sanity check for whether you’re overreacting or under-reacting to your own numbers.
A quick checklist for this week
- Add the current AI bot user-agents to robots.txt today, it takes five minutes and costs nothing.
- Turn on Cloudflare’s AI bot block if you’re on Cloudflare, or ask whoever manages your hosting whether an equivalent exists.
- Pull one month of server logs and look for crawler activity that sends zero traffic back, that’s your evidence, not a guess.
- Decide, on purpose, which bots you’re willing to let through because being cited matters to you, and which get blocked outright.
- Add an AI-training clause to your terms of service even though it’s not a fence, because it’s your paper trail if this ever escalates.
- Revisit the whole list every three months, because the bot names change faster than the advice does.
Frequently asked questions
Will blocking AI bots hurt my Google rankings?
No, not if you block them correctly. Google-Extended is separate from Googlebot, the crawler that indexes your site for search results. Blocking Google-Extended only stops your content being used for Gemini and AI Overviews training, it has no effect on your normal search ranking.
Can I really stop ChatGPT from using content I’ve already published?
Not retroactively. If a model was already trained on your content before you blocked its crawler, blocking now stops future scraping but doesn’t remove what’s already inside the model’s training data. There’s currently no reliable way to force removal of already-trained data from a model.
Is robots.txt legally enforceable?
No. Robots.txt is a voluntary standard, not a law. Reputable AI companies generally respect it because ignoring it is reputationally risky for them, but there’s no legal penalty built into the standard itself. Pairing it with clear terms of service wording gives you a stronger position if you ever need to argue non-consent formally.
What’s the single most effective thing I can do this week?
Turn on your CDN’s AI bot blocking feature if you have one, such as Cloudflare’s one-click block, because it enforces the block at the network level rather than relying on the bot choosing to obey robots.txt. It’s the closest thing to an actual lock rather than a polite sign on the door.
