Server logs: the only first-party evidence about AI crawlers

The vendor documentation tells you which user agent tokens exist and what each one controls, token by token, in AI crawlers and what each token controls. It does not tell you whether any of them ever visited your site. That answer exists in one place: your own request log.

Why logs carry more weight here than in classic SEO

For Google you have a first-party report. The Search Console crawl stats report shows requests by response code, by file type, by purpose and by Googlebot type, drawn from Google's own systems. No equivalent exists for ChatGPT, Claude or Perplexity. None of those companies publishes a site owner console, a crawl report or an impressions figure. What each engine does and does not expose is compared in how the answer engines differ.

So when an OpenAI, Anthropic or Perplexity agent fetches a page, the HTTP request it makes to your server, and the line your server writes about it, are the only first-party evidence you will ever get that those systems touched your content. Everything else on offer is sampled from the outside.

What a request line contains

Log formats differ, but the fields that matter for this work are the same everywhere.

Timestamp. Lets you group activity by day and see whether it changes after you publish or update a page.
Path. Which URL was requested. This is what turns a traffic total into a per-page picture.
Status code. Whether the fetcher received your content. A 200 means it did. A 403, 429, 404 or 5xx means it did not.
User agent. The User-Agent string the client sent. This is the only field that names the fetcher, and it is also the field that can be faked.
Bytes sent. A page that returns 200 with a body far smaller than the real article is usually a challenge page or an error template, not your content.
Referrer. Present on some requests and absent on most bot traffic. Useful when it appears, not something to build a method on.

If a CDN sits in front of your application, your origin log only records what the CDN passed through. Requests the edge blocked never reach you, which is covered in CDN and firewall rules.

Three kinds of AI fetch, and why the distinction is the point

The documented tokens divide into three roles, and a log filter that lumps them together answers nothing useful.

Training crawls collect content that may be used to train future models. Seeing them tells you your content is reachable. It tells you nothing about visibility in answers.

Search-index crawls build the retrieval index an engine cites from. These are the fetches that decide whether you are eligible to appear in an answer at all.

User-triggered fetches happen because a person asked a question at that moment and the engine went to get your page. This is the closest thing to a real-time signal that exists outside the engines. It does not prove you were quoted, but it does mean a live query routed a fetcher to your URL.

A method you can run on your own logs

Filter by the documented user agent substrings. Take the tokens from each vendor's own documentation and match on them: OpenAI, Anthropic, Perplexity and Google. Keep each token in its own bucket rather than merging them into an "AI traffic" total.
Group by path and by day. Per path, so you know which pages are being read. Per day, so you have a baseline to compare against later.
Read the three things logs actually tell you. Which pages get fetched. Which pages return something other than 200. Whether fetch activity changes after you publish or refresh a page.

That third point is the one most sites never check, and it is the cheapest experiment available: update a page, then watch whether the fetchers come back for it.

Verifying that a fetcher is who it claims to be

A User-Agent string is set by the client and can say anything. Any script can call itself GPTBot. Two techniques exist for confirming identity, and the companies support them unevenly.

Reverse DNS lookup resolves the requesting IP address to a hostname, checks that the hostname belongs to the expected domain, then resolves that hostname forward and confirms it returns the original IP. Published IP range matching compares the requesting IP address against a list of ranges the company publishes and maintains.

Google documents both. Its verification guidance describes the manual reverse DNS method, requiring the hostname to end in googlebot.com, google.com or googleusercontent.com, and publishes IP ranges in CIDR format as JSON files for common crawlers, special-case crawlers and user-triggered fetchers.

OpenAI publishes IP addresses per bot in its crawler documentation, at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. It does not document a reverse DNS method.

Anthropic publishes a single JSON list of crawler source addresses at claude.com/crawling/bots.json, stating in its crawler documentation that a source IP on that list indicates the crawler is coming from Anthropic. It does not document a reverse DNS method.

Perplexity publishes IP addresses at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, and its bot documentation asks that the published endpoints be treated as the source of truth because the addresses change. It does not document a reverse DNS method.

So published-range matching works for all four companies, and reverse DNS is documented only by Google. Do not invent a verification method for a company that does not publish one, and do not treat an unverified user agent match as proof of anything.

Cadence and honest limits

Monthly is the right rhythm for most sites. Day-to-day counts move for reasons that have nothing to do with you, and the questions logs answer well are structural: are the retrieval crawlers reaching my important pages, and is anything returning a status code other than 200.

The limits are real and worth stating plainly. A log line proves a fetch, not a citation. A fetch does not mean your page was used in an answer, because the engine may have retrieved it and discarded it. The absence of a fetch does not mean you were not used, because many answers are composed from an index built weeks earlier without any live request to your server. Logs are the ground truth for access. For what appears in answers, the constraints are set out in measuring AI search visibility honestly.