AI crawlers: who fetches your site and what each token controls
Every AI engine that answers questions about your market fetches web content, and each vendor publishes user agent tokens you can allow or block in robots.txt. The tokens are not interchangeable. Some feed model training, some feed the search index an engine cites from, and some fetch a page live because a user just asked about it. Block the wrong one and you give up visibility without protecting anything.
Three kinds of fetchers
The vendor documentation separates cleanly into three roles. Training crawlers collect content that may be used to train future foundation models. Search crawlers build the retrieval index an engine uses to ground and cite its answers. User-triggered fetchers visit a page at the moment a user's question requires it. Only the first role touches training. Only the other two decide whether you can appear in answers.
OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
OpenAI documents its bots along exactly these lines. GPTBot crawls content for training generative AI foundation models, and disallowing it indicates your content should not be used in that training. OAI-SearchBot is used to surface websites in ChatGPT's search features, and OpenAI states that sites opted out of it will not be shown in ChatGPT search answers. ChatGPT-User performs fetches for user actions in ChatGPT and Custom GPTs; because these are user initiated, OpenAI notes that robots.txt rules may not apply, and this agent does not determine search inclusion. The practical point: blocking GPTBot does not remove you from ChatGPT search. Blocking OAI-SearchBot does.
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic documents three agents with the same division of labor. ClaudeBot collects web content that could contribute to model training, and disallowing it signals that your future material should be excluded from training datasets. Claude-SearchBot analyzes content to improve the relevance and accuracy of search responses; blocking it may reduce your visibility in those results. Claude-User fetches pages when individuals ask Claude questions, and blocking it prevents your content from being retrieved for a user's query.
Perplexity: PerplexityBot and Perplexity-User
Perplexity's crawler documentation states that PerplexityBot exists to surface and link websites in Perplexity's search results and is not used to crawl content for AI foundation models. Blocking it therefore prevents nothing on the training side; it only costs you presence in Perplexity's results. Perplexity-User supports user actions and generally ignores robots.txt rules because a user requested the fetch. It is likewise not used for training.
Google: Googlebot and Google-Extended
Google-Extended is the most misread token on this list. Per Google's crawler documentation, it is not a separate crawler at all: crawling happens under existing Google user agents, and Google-Extended is a robots.txt token used in a control capacity. It controls whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google states explicitly that it does not affect a site's inclusion in Google Search and is not a ranking signal. AI features inside Search are governed by the normal Search controls that apply to Googlebot, not by Google-Extended. Blocking Google-Extended keeps your content out of Gemini training. Blocking Googlebot removes you from Search and everything built on it.
The trade-off, token by token
| Token | Blocking it stops | Blocking it costs you |
|---|---|---|
| GPTBot | Use of your content in OpenAI model training | No documented effect on ChatGPT search |
| OAI-SearchBot | Inclusion in ChatGPT search | Being shown in ChatGPT search answers |
| ChatGPT-User | Little; robots.txt may not apply to user actions | User-initiated fetches of your pages |
| ClaudeBot | Use of your content in Anthropic model training | Nothing documented on the search side |
| Claude-SearchBot | Indexing for Claude's search responses | Visibility in those responses |
| Claude-User | Retrieval for individual user queries | Visibility in user-directed web search |
| PerplexityBot | Nothing on training; it is a search bot | Appearing in Perplexity results |
| Google-Extended | Gemini training and the listed grounding uses | Nothing in Google Search, per Google |
| Googlebot | Google Search itself | Search visibility and the AI features built on it |
The clean way to decide: training tokens are a policy choice you can make either way, while search and user tokens are a revenue choice, because an engine that cannot retrieve you cannot cite you. Verifying that the retrieval crawlers can actually reach your pages belongs to technical SEO, and what they do with your content once they have it is covered in what is GEO.