Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
Does llms.txt actually do anything?
The honest answer has two halves, and the half nobody publishes is the one that decides it. Here is what is verifiable, what is not, and the test that would settle it for your own domain.
llms.txt is a markdown file at the root of a domain that describes what a site contains and where the important pages are. It is a content map. Two things it is not: it is not a permissions file, and it is not a standard any model provider has committed to consuming.
That second point is where the argument lives, and most writing on the subject picks a side without evidence. What follows is the part that is checkable.
What is actually known
| Sites publishing one, web-wide | 70,213 |
| Sites disallowing GPTBot, for scale | 107,182 |
| Scans here with no llms.txt | 18.3% |
| AI-visibility vendors publishing one | roughly one in three of 33 |
| Model providers documenting it as an input | none publicly |
Read the first two rows together. More sites have gone to the trouble of blocking a single AI crawler than have published a content map for all of them. And the third row is worth sitting with: two-thirds of the companies selling advice about AI visibility have not published the file on their own domain. That is not hypocrisy so much as a signal about how confident the people closest to it actually are.
What it cannot do
It cannot restrict any crawler. It grants no permissions and withholds none — that is robots.txt's job, and increasingly your edge's. It does not make an unreadable site readable. If your machine files are being served from a stale cache, or your homepage delivers 2% text in half a megabyte, an llms.txt describing that page changes nothing about what gets fetched.
It also cannot fix an entity problem. A file that names your business without stating hours, service area or price answers nothing anyone actually asked.
What it plausibly does
Three arguments survive scrutiny, in descending order of strength.
It is a cheap, well-formed statement of what you consider important. Whether or not a specific model reads the file, writing it forces a decision about which twenty pages matter, and that decision is reusable everywhere else.
It is machine-parseable prose in a world of markup. The median site in the vendor corpus delivers under one visible word per twenty bytes of payload. An llms.txt is almost pure text. If anything does read it, the cost per useful token is dramatically lower than fetching the pages it points at.
It costs close to nothing. A file, one deploy. Set against that: publishing one badly is a real defect. A stale or unparseable llms.txt puts a broken artefact on your domain and tells anyone measuring you that your machine layer is unmaintained.
The test that would settle it — and why almost nobody can run it
Every argument above is inference. There is one measurement that is not: do verified crawlers request the file?
That question cannot be answered by scanning a site from outside. It needs the server's own access logs, and it needs the requests in them checked against the IP ranges the operators publish, because a user-agent is a claim rather than an identity — 37% of the checkable crawler claims in this dataset came from outside the published range. Counting raw log lines that say GPTBot would answer the wrong question confidently.
Run properly, that log check tells you three things nobody can tell you in general: whether the file is fetched, by which verified operators, and how often relative to your sitemap. It is a per-domain answer, and per-domain is the only honest form this answer comes in.
The files it gets confused with
Three conventions are routinely discussed as if they were one thing. They are not, and they have different jobs.
| File | Job | Adoption, web-wide |
|---|---|---|
robots.txt | Permissions and sitemap discovery | near universal |
llms.txt | Content map: what is here and where | 70,213 |
agents.md | Instructions written to be obeyed by an agent | early, template-driven |
| UCP profile | Agentic-commerce discovery | 41,525 |
The third row carries a risk the first two do not. agents.md is the only file on a typical domain written specifically to be followed by a machine, and on many platforms it arrives by default rather than by authorship — whole hosting populations ship an identical template, including steering language directed at assistants, on domains whose owners have never opened the file. If something on your domain is written to instruct an agent, it is worth knowing whether you wrote it.
And whichever of these you publish, the same delivery problem applies to all of them: MACHINE_FILE_CACHE_STALE fires on 1.6% of scans and MACHINE_FILE_CACHE_SPLIT on 0.2% — cases where the file a crawler receives is not the file the origin holds. A machine file is not published until the bare URL returns it.
What to do
Publish one if it costs you an hour, keep it accurate, and check that it serves text/plain or text/markdown on the bare URL rather than a cached challenge page. Then stop thinking about it, because it is not the constraint.
The constraint is upstream: 30.9% of scans here serve crawlers a stale cached object, 13.7% deliver a page that is overwhelmingly code, and 3.4% refuse at least one answer engine outright. An llms.txt on a site with any of those is a note pinned to a locked door. Fix the door, publish the note, and treat anyone selling llms.txt as a growth strategy with the scepticism that claim has earned.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Is llms.txt worth it?
It costs about an hour and cannot hurt if kept accurate, but it is not the constraint on whether answer engines can use your site. No model provider publicly documents it as an input, roughly 70,213 sites publish one, and only about a third of 33 AI-visibility vendors publish one on their own domain.
Is llms.txt a standard?
It is a convention, not a ratified standard, and it is separate from robots.txt. It describes content and grants no permissions. Nothing in it can allow or block a crawler.
How would I know if anything reads my llms.txt?
Check your server access logs for requests to the file, and verify each request's source address against the operator's published IP ranges before counting it. Raw log lines naming a crawler are unreliable: 37% of checkable crawler claims in this dataset came from outside the published range.