AI Crawler User Agents: Which Bots to Allow, Which to Block

The vendors say so themselves. OpenAI says each of its settings is independent of the others, so a site can let OAI-SearchBot show it in ChatGPT search while telling GPTBot its content isn't for training. Here's every bot the main vendors list as of October 2026, what each one does, and what blocking it costs you.

Every AI bot by vendor, and what blocking it means

Bot token	Vendor	Kind	What it does	What blocking it does
GPTBot	OpenAI	Training	Crawls content that may be used to train OpenAI's generative AI foundation models	Signals your content shouldn't be used for that training
OAI-SearchBot	OpenAI	Search	Surfaces websites in ChatGPT's search features	You won't be shown in ChatGPT search answers, though you can still appear as navigational links
ChatGPT-User	OpenAI	User-triggered	May visit a page when a person asks ChatGPT or a Custom GPT a question	Robots.txt rules may not apply, and it isn't used to decide what appears in search
OAI-AdsBot	OpenAI	Ads check	Validates the safety of pages submitted as ads on ChatGPT, and only visits those pages	Its data isn't used to train foundation models
ClaudeBot	Anthropic	Training	Collects web content that could contribute to training	Signals your future materials should be left out of training datasets
Claude-SearchBot	Anthropic	Search	Improves search result quality	Prevents indexing for search and may reduce your visibility and accuracy in results
Claude-User	Anthropic	User-triggered	Fetches pages when a person asks Claude a question	May reduce your visibility for user-directed web search
Google-Extended	Google	Training and grounding token	Controls whether content Google crawls may train future Gemini models and be used for grounding	Doesn't affect inclusion in Google Search and isn't a ranking signal
PerplexityBot	Perplexity	Search	Surfaces and links websites in Perplexity search results, and Perplexity says it isn't used to crawl for AI foundation models	Can reduce your visibility in Perplexity answers
Perplexity-User	Perplexity	User-triggered	May visit a page when a person asks a question, and isn't used for crawling or training	Generally ignores robots.txt, because a user asked for the fetch

Training, search and user-triggered: three separate decisions

Training crawlers are about whether your words help train future models. Blocking GPTBot, ClaudeBot or Google-Extended is a statement about training, and the vendors' own descriptions treat it that way. Google goes further: it says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It also has no separate HTTP user agent string, so you won't spot it in your logs. It's a token you name in robots.txt.

Search crawlers are how you get into the answers. Blocking OAI-SearchBot means ChatGPT search won't show you, and disabling Claude-SearchBot stops Anthropic indexing you for search. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.

User-triggered fetchers only visit when a person asks a question. They aren't crawling the web on their own, and because a human started the request, OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots respect do-not-crawl signals in robots.txt and anti-circumvention technologies such as CAPTCHAs, and that disabling Claude-User may reduce visibility for user-directed web search.

Google lists more crawlers than most people expect. Googlebot rules affect Google Search, including Discover and every Search feature, plus Google Images, Video mode and Google News. Google-CloudVertexBot covers crawls site owners request for Vertex AI Agents, and GoogleOther is a generic research crawler tied to no specific product.

Writing robots.txt groups for AI bots

Robots.txt lives in the top-level directory of your site, and Anthropic says to repeat your rules on every subdomain. Each bot gets its own group. To block one from the whole site, the group is two lines, one under the other: User-agent: ClaudeBot and Disallow: / with nothing after the slash. Swap the name for any token in the table. Anthropic also supports the non-standard Crawl-delay extension, for example Crawl-delay: 1, if you'd rather slow its bot down than shut it out.

1. Decide on training first: GPTBot, ClaudeBot and Google-Extended.
2. Decide on search visibility separately: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
3. Open robots.txt in the top-level directory of each domain and subdomain.
4. Add one group for each bot you want to restrict, with its User-agent line and a Disallow line.
5. Leave IP blocking out of it, because it can stop a vendor reading your robots.txt.
6. Let the search bots through your firewall using each vendor's published IP ranges.
7. Give it about a day, then check your server logs.

Three traps that break your opt-out or your visibility

Anthropic warns that blocking by IP address may not work correctly or persistently, because it stops Anthropic reading your robots.txt. A firewall block can stop the bot seeing the very file that says no. Opting out requires editing robots.txt.

The opposite problem hits sites that want to be found. If a firewall blocks the bots, Perplexity says you may need to allow each one by matching its user agent to its own published IP ranges. OpenAI publishes IP lists for each of its bots as searchbot.json, gptbot.json, chatgpt-user.json and adsbot.json on openai.com. In Cloudflare, Bot Fight Mode can't be skipped with WAF custom rules and can still challenge Perplexity's bots, so check it if those bots never show up.

Don't trust a user agent string on its own either. Google cautions that it can be spoofed and points to its guide on verifying whether a visitor really is a Google crawler. Google's strings carry a Chrome version number that changes, so use a wildcard for the version when you filter logs. And for ChatGPT search, OpenAI says a robots.txt change can take about 24 hours to take effect, so don't expect to see a difference within the hour.

What changed on 30 October 2026

Two updates landed on the same day. Google's common crawlers list was last updated on 30 October 2026, and it says Google intends to support the AIPREF shared, machine-readable training preference signal, written user-agent: __usage_ai-training, in early 2027.

OpenAI's news feed lists a 30 October 2026 item called 'Making it easier for websites to express AI training preferences'. Its one-line description says OpenAI will honour a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers. Until a shared signal is in place, the per-bot groups above are what works.

My default for a business that wants to be cited

If you want AI tools to find and cite you, I'd allow the search bots and the user-triggered fetchers, and make the training call on your own terms. Blocking GPTBot, ClaudeBot or Google-Extended is a fair choice if you don't want your content in training data, and by the vendors' own descriptions it isn't what decides whether you show up in their answers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot is the one I'd avoid, because that's the visibility you're after.

Two honest limits. Robots.txt is a request, not a lock, and the user-triggered fetchers may ignore it. And none of this is a legal or technical guarantee. If you need content kept out of reach, that's a different job, covered in this guide to stopping AI bots scraping your content. If your question is how soon a change shows up, how often AI crawlers recrawl a website covers the timing.

If you want your business found and cited in AI answers, and you'd like the bot rules set with that goal in mind, that's part of the GEO work I do. I run AI training and workshops for teams, and help businesses with AI enablement, AI automation and GEO, which is about getting your business seen and cited in AI answers. Most of that work is with SaaS marketing teams, telecom companies and small businesses. Here's how we can work together.

Frequently asked questions

What's the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI's foundation models. OAI-SearchBot surfaces websites in ChatGPT search. They're set independently, so you can allow one and block the other.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It controls whether content Google crawls may be used to train future Gemini models and for grounding.

Do AI bots have to obey robots.txt?

Not all of them. Anthropic says its bots respect robots.txt, but OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it because a person asked for the fetch.

How long does a robots.txt change take for ChatGPT search?

OpenAI says a robots.txt change can take about 24 hours to take effect for search.

Can I block AI bots by IP address instead?

Anthropic warns that IP blocking may not work correctly or persistently, because it stops Anthropic reading your robots.txt. Opting out means editing robots.txt.