ChatGPT, Perplexity, Gemini, Claude and Copilot: where each engine gets its answers

Answer engines look similar from the outside: a question goes in, a cited answer comes out. Underneath they differ in three ways: the index each engine retrieves from, the crawlers that feed that index, and how citations reach the reader. Most practical differences follow from the index, because an engine can only cite what its retrieval system can reach.

One caveat up front. Not every backend is public. Some companies document their retrieval pipeline in detail, others document only their crawlers. Where something is not documented, this page says exactly that. The shared mechanics of retrieval-augmented generation are covered in how answer engines choose their sources; this page is the per-engine reference.

ChatGPT

OpenAI documents three fetchers: GPTBot collects content for training foundation models, OAI-SearchBot surfaces websites in ChatGPT's search features, and ChatGPT-User fetches a page when a user's action requires it. The documented lever is OAI-SearchBot: OpenAI states that sites opted out of it will not be shown in ChatGPT search answers. Token details and blocking trade-offs are covered in AI crawlers.

What OpenAI does not document is the composition of the index behind ChatGPT search or how sources are ranked inside it. Claims that it runs on any particular third-party index are not confirmed by OpenAI and should be treated as unverified. OpenAI also documents no publisher-facing citation report.

Perplexity

Perplexity documents two agents. PerplexityBot surfaces and links websites in Perplexity's search results and, per the documentation, is not used to crawl content for AI foundation models. Perplexity-User fetches pages for user questions and generally ignores robots.txt because a person requested the fetch. Blocking PerplexityBot therefore protects nothing on the training side; it only removes you from Perplexity's results.

Beyond those crawler descriptions, Perplexity does not publicly document how its retrieval index is assembled or how sources are ranked, and it documents no publisher-facing citation report.

Gemini and Google's AI features

Google is the most fully documented engine. Its AI features guide states that AI Overviews and AI Mode use retrieval-augmented generation grounded in the core Search ranking systems, retrieving pages from the Search index. The entry requirement is classic: crawlable, indexed, snippet-eligible. Answers show prominent, clickable links to the supporting pages, and Search Console includes a Generative AI performance report, the first-party way to see this traffic.

On the crawler side, Googlebot feeds Search and everything built on it, while Google-Extended is a robots.txt control token that governs Gemini model training and specific grounding uses, not inclusion in Search. The distinction is commonly misread.

Claude

Anthropic documents three agents: ClaudeBot collects web content that could contribute to model training, Claude-SearchBot analyzes content to improve the relevance and accuracy of search responses, and Claude-User fetches pages when individuals ask Claude questions. Anthropic's developer documentation states that answers produced with web search include citations for the sources drawn from the results.

The index behind that search is not publicly documented. The crawler article describes what each agent does, not where search results come from, and the developer documentation names no search provider or index. Any claim about which index backs Claude's web search is unverified. Anthropic documents no publisher-facing citation report.

Microsoft Copilot

Copilot is the clearest case of an engine built on an existing search index. Microsoft documents the retrieval path for Microsoft 365 Copilot and Copilot Chat: when web search is enabled, Copilot generates a search query from the user's prompt, sends it to the Bing search service, and uses the results to ground the response in web data. Visibility in those answers therefore runs through Bing's index and its crawler, Bingbot. How Copilot weighs or selects among the Bing results it receives is not documented.

In February 2026, Bing Webmaster Tools added an AI Performance report in public preview, covering citations across Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations. It shows which URLs are cited, the grounding queries that retrieved them, and how citation activity changes over time. That makes Google and Microsoft the only two companies with a first-party citation report; OpenAI, Anthropic and Perplexity document no equivalent.

What this means for your robots.txt

The tokens above divide into training crawlers, search crawlers and user-triggered fetchers, and blocking them has different consequences. Blocking a training bot such as GPTBot or ClaudeBot is a policy choice with no documented cost in any engine's answers. Blocking a search bot such as OAI-SearchBot or PerplexityBot removes you from that engine's citation surface. The trade-offs, token by token, live in AI crawlers.

The work is shared, the surfaces differ

Nothing above adds up to five separate disciplines. Every engine needs the same inputs: pages its crawler can reach, passages that state an answer plainly enough to quote, and content that is kept current. The engines differ in how hard they lean on freshness: in a study of 7,683 pages and 47,097 citations, Gemini cited content updated within the past year 78 percent of the time, ChatGPT 73 percent and Perplexity 65 percent. What to update, and how often, is covered in freshness.

What does differ is your standing per engine, because the indexes differ. The only way to know where you stand is to measure per engine, with repeated sampling rather than spot checks. Sample across languages too: in measurements of LLM brand responses, query language explains 26.5 to 32.0 percent of answer variance, while brand identity explains 1.5 percent. What that means for markets is covered in multilingual GEO.