llms.txt: the proposal, the promise and the measured reality

llms.txt is a proposed convention: a markdown file served at /llms.txt that gives language models a curated map of a site's most useful content. It is not a standard, and no major engine has adopted it. That has not stopped a small industry from selling llms.txt generation as an AI visibility lever, so this page checks the claim against the proposal itself, the vendor documentation and the server logs.

Where the proposal comes from

The proposal was published by Jeremy Howard in September 2024 at llmstxt.org. Its reasoning is practical. Language model context windows are small relative to a whole website, and HTML pages carry navigation, scripts and markup that waste that space. A curated markdown index would let a model or an agent find the substantive material directly.

The specification is short. It asks for a markdown file at the root path with an H1 naming the site, the only required element, followed by a blockquote summary, optional prose sections, and H2-delimited lists of links with short notes. A section titled "Optional" marks links that can be skipped when context is tight. The proposal also recommends serving clean markdown copies of important pages at the same URL with .md appended. None of this is hard to produce, and none of it is the problem. The problem is on the consumption side.

Who says they use it

No major engine documents honoring llms.txt for retrieval. Google states directly in its AI features guidance that llms.txt is ignored by Google Search; the rest of that document is walked through in what Google actually says about AI search optimization.

OpenAI, Anthropic and Perplexity each publish detailed documentation for their crawlers, covered token by token in AI crawlers and what each token controls. None of that documentation mentions llms.txt. There is no statement from any of these vendors committing a production answer system to reading the file. The support story is not mixed or emerging. As of this writing, it is absent.

What the logs show

Documentation could lag behind practice, so the stronger evidence is server logs. If engines used the file, they would fetch it. Two measurements say they do not.

A 90-day crawler-log experiment recorded 62,100 AI-bot requests. Of those, 84 fetched the llms.txt file, about 0.1 percent.

A study of 137,000 sites measured a one-month window and found that 28 percent of the domains published a valid llms.txt file, and that 97 percent of those files were never fetched by anything, AI crawler or otherwise. The same data shows the other side of the coin: on domains without the file, AI bots made zero requests for it. They do not go looking.

A file that is not fetched cannot influence what an engine says about you. Whatever llms.txt may become, the claim that it improves AI visibility today has no measured support.

llms.txt is not robots.txt

The two files are often bundled in the same advice, and they do different jobs. robots.txt is an access control convention: the AI vendors document their user agent tokens for it, request it routinely and state what each token governs. llms.txt is a content discovery proposal that those same vendors have not adopted. Nothing written in an llms.txt file grants, restricts or signals anything about access. Decisions about which AI systems may fetch your site are made in robots.txt, token by token, as covered in AI crawlers. Confusing the two files leads to wrong decisions in both directions.

A reasonable position

Publishing an llms.txt file costs a few minutes and nothing after that. Google says the file neither helps nor harms. If adoption ever arrives, a published file is already in place, and your own server logs will show the day fetch rates change. On that basis, having one is a harmless hedge.

Paying for one is a different matter. A file that engines do not read cannot be an optimization, and money spent generating, auditing or monitoring llms.txt is money not spent on the unglamorous work that is documented to matter: indexable pages, answerable content, verifiable claims. Vupie generates an llms.txt for its own site and for customer sites and claims nothing for it, which is the stance the evidence supports.