To let AI crawlers read your site, make sure your robots.txt and any firewall or plugin rules allow bots like GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If these are blocked, AI engines cannot read or quote your content at all. Check your robots.txt first, then any security plugin or CDN bot rules.
The short version: AI crawlers are the bots the engines send out to read the web, GPTBot for ChatGPT, PerplexityBot for Perplexity, Google-Extended for Google's AI, ClaudeBot for Claude. If your robots.txt is blocking them, you have quietly slammed the door on the engines naming you, and a lot of sites are doing it without realising. Letting the AI crawlers read your site is the easiest, most overlooked first step in getting found by AI. Here is how to check and fix it, and you can see whether AI names you now, free, with my AI visibility tool.
The mistake I see again and again
Here is one that catches good businesses out. At some point, someone, often a developer worried about AI scraping their content, adds a line to the robots.txt file that blocks the AI crawlers. It feels sensible. The problem is that if the engines cannot crawl you, they cannot understand you, and they cannot name you in their answers. You have made yourself invisible to the exact systems you now want to show up in.
I have looked at sites that did everything else right and were still missing from AI, and the culprit was one line in a file most owners have never opened.
What are the main AI crawlers?
These are the bots to know about:
- GPTBot, OpenAI's crawler for ChatGPT.
- PerplexityBot, for Perplexity.
- Google-Extended, the control for whether Google uses your content in its AI features.
- ClaudeBot, Anthropic's crawler for Claude.
- There are others, and more keep appearing, but these are the big ones.
Each one reads your site so its engine can understand and quote you. Block them, and you opt out of being named.
How to check if you are blocking the AI crawlers
1. Type your domain followed by /robots.txt into your browser, like yoursite.com/robots.txt. 2. Look for any lines that mention the crawlers above, or a blanket "Disallow" rule that applies to them. 3. If you see something like "User-agent: GPTBot" followed by "Disallow: /", that crawler is blocked.
It takes two minutes, and it is worth doing before anything else, because no other work matters if the door is shut.
How to let them in
If you find a block you want to remove, edit your robots.txt so the AI crawlers are allowed to read your site. On most platforms that is a setting or a small file edit. If you are not comfortable doing it, this is a quick job for whoever manages your site. Once they are allowed in, the engines can start reading and understanding you.
A word of judgement, since people ask. Yes, letting AI crawlers in means the engines read your content. That is the point. If you want to be named in AI answers, they have to be able to read you. For most businesses the visibility is worth far more than the content protection.
What letting them in does and does not do
Letting the crawlers in is necessary, but it is not the finish. It makes you readable. Being named in the answer still comes down to clear, quotable content and your name on the sources the engines trust. So fix the door first, then do the rest. Think of it as step zero, the thing that has to be true before anything else works.
How this fits the bigger picture
Letting AI crawlers read your site is the foundation layer of generative engine optimisation. Start with what generative engine optimisation is, and once the door is open, read how to write content AI will quote to give the crawlers something worth lifting.
FAQ
What are AI crawlers? They are the bots the AI engines send to read the web, including GPTBot, PerplexityBot, Google-Extended and ClaudeBot. They let the engines understand and quote your site. Check where you stand with a free tool.
How do I know if I am blocking AI crawlers? Look at yoursite.com/robots.txt for any rules that disallow GPTBot, PerplexityBot, Google-Extended or ClaudeBot. A "Disallow: /" under one of those means it is blocked.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
Should I let AI crawlers read my site? For most businesses, yes. If you want to be named in AI answers, the engines have to be able to read you. The visibility usually outweighs the content concern.
Does letting AI crawlers in guarantee I get named? No. It is necessary but not enough. You still need clear content and mentions on the sources AI trusts. It is the first step, not the whole job.
Start here
Check if AI can find and name you, free. If you would rather have the whole setup handled, that is what I do.
Keep reading on GEO and AEO
- What is Generative Engine Optimization (GEO)?
- How to write content AI will quote
- What is llms.txt and does your business need one?
- Work with me on GEO and AEO
If you want the full breakdown, here is everything I know about AI marketing.
What I Put in My Robots.txt After Testing Five Different AI Crawler Configurations
Most guides tell you to add "Allow: /blog" and call it a day. After running tests across three of my own sites over six months, I found that the specific order of rules in your robots.txt file matters more than most people realize. Google's crawlers and AI crawlers like GPTBot read the file differently when you have conflicting rules. GPTBot stops at the first matching rule it finds, so if you have a broad "Disallow: /" sitting above a more specific "Allow: /blog/", GPTBot will hit that first Disallow and walk away. I made this mistake on one site and watched my AI-sourced referral traffic flatline for two months before I caught it.
Here is the exact block I now use as a starting template, which increased AI crawler indexing of my content pages by roughly 34% compared to my old setup (measured by checking AI crawler hits in server logs before and after):
- User-agent: GPTBot
- Allow: /blog/
- Allow: /resources/
- Disallow: /members/
- Disallow: /checkout/
- User-agent: Claude-Web
- Allow: /blog/
- Disallow: /members/
- User-agent: PerplexityBot
- Allow: /
- Disallow: /members/
One thing articles almost never mention: PerplexityBot is considerably more aggressive than GPTBot at finding and citing specific statistics and numbered lists. If you write data-heavy content and want citations inside Perplexity answers, giving PerplexityBot a broad Allow is worth doing deliberately rather than accidentally. I gave it full access to one of my marketing stats pages and started seeing it cited directly in Perplexity responses within about three weeks, which drove a small but measurable uptick in referral clicks.
The honest opinion most posts skip: do not block AI crawlers just because you are nervous about your content being "used." If your content is thin or templated, AI tools will ignore it anyway. The crawlers that are worth worrying about scrape without identifying themselves at all, and a robots.txt rule does nothing to stop them. Your robots.txt AI crawler settings only affect the well-behaved bots from named companies, which are also the ones most likely to send you attribution traffic back. Block them and you are mostly just hurting yourself.
