CDN and firewall rules: the invisible reason engines cannot read you

A site can publish a permissive robots.txt, return clean status codes and still be unreadable to every answer engine except Google. The cause usually sits one layer below the application: a CDN, a web application firewall or a bot management product that refuses the request before your server ever sees it. A robots.txt audit cannot detect this, because robots.txt is not the layer doing the blocking.

Two layers that can disagree

robots.txt is a request you publish. A compliant crawler reads it and honors it voluntarily, and nothing enforces it. A CDN, WAF or bot management product does the opposite: it decides access at the edge, before the request reaches your application, and it does not consult your robots.txt when deciding. When the two layers disagree, the infrastructure layer wins. Your robots.txt can say Allow: / for every documented AI token while the edge returns 403 to all of them.

The common failure modes

• Managed bot rules. Bot management products classify clients and apply an action per class. Automated clients that the product does not recognize, or recognizes but categorizes as unwanted, get challenged or blocked. A newly documented AI user agent is exactly the kind of client that lands in an unrecognized bucket.
• Rate limiting. A crawler fetching a few hundred pages in a short window looks like a burst. Thresholds tuned for human traffic turn a normal crawl into a run of 429 responses, so the engine retrieves a fraction of your site and stops.
• Geographic or ASN blocking. Rules that block traffic from countries you do not sell to, or from cloud provider networks, also block fetchers that originate from those networks. AI crawlers run in data centers, not on residential connections.
• JavaScript challenges and CAPTCHA interstitials. A fetcher that does not execute JavaScript cannot solve the challenge. It receives the challenge page instead of your content, frequently with a 200 status code, so the response looks successful in a status-code report while carrying none of your text.
• Defaults nobody chose. Hosting platforms and CDNs ship protections that are on before the site owner opens the settings. The configuration was never a decision, so it is never reviewed.

Why this stays invisible

Your browser test passes, because your browser executes the challenge script and solves it. Search Console keeps reporting normally, because bot products verify and allow Googlebot by default. So the one engine with a first-party report is the one engine that is not blocked, and every other engine gets nothing without ever telling you.

How to diagnose it

1. Request your own pages with each documented AI user agent. Send the User-Agent header set to each token published by OpenAI, Anthropic, Perplexity and Google. Do this from outside your own network, because your office or server IP address is often allowlisted.
2. Compare status code and body length against a normal browser request. The status code alone is not enough. A challenge interstitial can return 200 with a body a fraction of the real page size, so record both numbers for every token and compare them to the browser baseline.
3. Confirm against your logs. If a request was blocked at the edge, it will appear in the CDN log and be absent from the origin log. That difference tells you which layer acted. The field-by-field method is in server logs and AI bots.

What the providers document

Cloudflare's AI Crawl Control documentation gives three actions per crawler: allow, block, and charge. It states that blocking a crawler creates or updates a WAF custom rule on your zone to enforce the block, and lets you choose whether the response is 403 or 402.

The defaults are the load-bearing part. On July 1, 2025 Cloudflare launched a one-click block option and a pay-per-crawl marketplace. In its July 1, 2026 announcement it split AI traffic into three classifications, Search, Agent and Training, and stated that from September 15, 2026, for all new domains onboarding to Cloudflare, Training and Agent are blocked by default on pages that display ads, while Search remains allowed by default. Owners who do not want the new defaults can change them in their security settings before that date. The same post notes that multi-purpose crawlers such as Googlebot, Applebot and BingBot are blocked for customers who block Training, because a crawler that combines behaviors is evaluated against all of them and the most restrictive applicable rule is enforced. If your site is on Cloudflare, this is a setting you should read rather than assume.

AWS documents a CategoryAI rule in the AWS WAF Bot Control rule group, with a rule action of Block, and states that this rule applies the action to all matches regardless of whether the bots are verified or unverified. That is stricter than the other category rules, which act only on unverified bots. The rule group is something you add to a web ACL, so its effect depends on whether it has been added and at which protection level.

Fastly documents AI classification in its bot management guidance, exposing fastly.bot.category.is_ai_crawler and fastly.bot.category.is_ai_fetcher so that policy is written explicitly rather than inherited. Where a provider does not publish its default posture, do not assume one. Check the configuration in the dashboard and confirm it with the test above.

Make the decision, do not inherit it

Blocking training crawlers and blocking search or answer crawlers are two different decisions with two different costs, set out token by token in AI crawlers and what each token controls. Both positions are defensible. What is not defensible is arriving at either one by accident, through a default you never saw, a rate limit tuned for something else, or a bot rule that treats every unfamiliar client as hostile. An engine that cannot fetch your page cannot cite it, and the infrastructure layer is where that outcome is actually decided.

What to verify after any infrastructure change

1. Fetch a representative page with each documented AI user agent from an external network and record status code and body length.
2. Compare those numbers with a normal browser request for the same URL.
3. Check that no challenge, CAPTCHA or interstitial is served to non-rendering clients.
4. Review the AI or bot category settings in your CDN dashboard and confirm they match your intended policy.
5. Confirm rate limits allow a full crawl of your site at a reasonable pace.
6. Watch the logs for a week and check that fetch volume and status codes did not change.