Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 2 views

Does llms.txt actually do anything?

The honest answer has two halves, and the half nobody publishes is the one that decides it. Here is what is verifiable, what is not, and the test that would settle it for your own domain.

llms.txt is a markdown file at the root of a domain that describes what a site contains and where the important pages are. It is a content map. Two things it is not: it is not a permissions file, and it is not a standard any model provider has committed to consuming.

That second point is where the argument lives, and most writing on the subject picks a side without evidence. What follows is the part that is checkable.

What is actually known #

Read the first two rows together. More sites have gone to the trouble of blocking a single AI crawler than have published a content map for all of them. And the third row is worth sitting with: two-thirds of the companies selling advice about AI visibility have not published the file on their own domain. That is not hypocrisy so much as a signal about how confident the people closest to it actually are.

What it cannot do #

It cannot restrict any crawler. It grants no permissions and withholds none — that is robots.txt's job, and increasingly your edge's. It does not make an unreadable site readable. If your machine files are being served from a stale cache, or your homepage delivers 2% text in half a megabyte, an llms.txt describing that page changes nothing about what gets fetched.

It also cannot fix an entity problem. A file that names your business without stating hours, service area or price answers nothing anyone actually asked.

What it plausibly does #

Three arguments survive scrutiny, in descending order of strength.

It is a cheap, well-formed statement of what you consider important. Whether or not a specific model reads the file, writing it forces a decision about which twenty pages matter, and that decision is reusable everywhere else.

It is machine-parseable prose in a world of markup. The median site in the vendor corpus delivers under one visible word per twenty bytes of payload. An llms.txt is almost pure text. If anything does read it, the cost per useful token is dramatically lower than fetching the pages it points at.

It costs close to nothing. A file, one deploy. Set against that: publishing one badly is a real defect. A stale or unparseable llms.txt puts a broken artefact on your domain and tells anyone measuring you that your machine layer is unmaintained.

The test that would settle it — and why almost nobody can run it #

Every argument above is inference. There is one measurement that is not: do verified crawlers request the file?

That question cannot be answered by scanning a site from outside. It needs the server's own access logs, and it needs the requests in them checked against the IP ranges the operators publish, because a user-agent is a claim rather than an identity — 37% of the checkable crawler claims in this dataset came from outside the published range. Counting raw log lines that say GPTBot would answer the wrong question confidently.

Run properly, that log check tells you three things nobody can tell you in general: whether the file is fetched, by which verified operators, and how often relative to your sitemap. It is a per-domain answer, and per-domain is the only honest form this answer comes in.

The files it gets confused with #

Three conventions are routinely discussed as if they were one thing. They are not, and they have different jobs.

The third row carries a risk the first two do not. agents.md is the only file on a typical domain written specifically to be followed by a machine, and on many platforms it arrives by default rather than by authorship — whole hosting populations ship an identical template, including steering language directed at assistants, on domains whose owners have never opened the file. If something on your domain is written to instruct an agent, it is worth knowing whether you wrote it.

And whichever of these you publish, the same delivery problem applies to all of them: MACHINE_FILE_CACHE_STALE fires on 0.5% of scans and MACHINE_FILE_CACHE_SPLIT on 0.1% — cases where the file a crawler receives is not the file the origin holds. A machine file is not published until the bare URL returns it.

The delivery problem is measurable even though adoption is not #

There is a way around the evidence gap in this argument, and it is worth setting out because it changes what you should spend the hour on.

Too few sites publish an llms.txt to measure how well it is served. But nearly every site publishes a robots.txt, and the two files are delivered by the same stack — same origin, same cache rules, same content-type configuration, same bot defence. So the failure rates on robots.txt are a reasonable forecast of the failure rates your llms.txt will have, because none of those failures are properties of the file.

Read the first two rows together and the point lands. The most-published machine file on the web — the one with three decades of tooling and documentation behind it — still fails to arrive intact on a measurable share of sites. A file with none of that ecosystem, written once and rarely revisited, will not do better.

Which reframes the whole question. The interesting variable is not whether operators will adopt llms.txt. It is whether your stack can deliver a plain text file to a non-browser client at all, and that is something you can find out this afternoon regardless of what any operator decides.

What to put in it, judged by what a client could use #

The format is deliberately plain: a heading with the site name, a short blockquote saying what the site is, then sections of links with a one-line description under each.

The descriptions are the part people skip and the only part with any information in them. A bare list of URLs tells a client nothing it could not get from your sitemap, which it already has. A line saying what each page answers is the entire value of the file — it is the one place you get to state, in prose, what your pages are for.

Length works against you. A file listing forty pages says nothing about priority, which makes it a sitemap in markdown. Ten to twenty entries, each one a page you would be happy to have quoted back at you, is the shape that carries a signal. Everything you add past that point dilutes the ones you care about.

And every URL in it has to resolve. A curated index that points at three moved pages is worse than no index, because it is a statement about your own site that is demonstrably wrong — made in a file whose whole purpose is to be trusted about your own site.

The cost and the risk, priced honestly #

An hour to write, a few minutes to verify, and close to nothing to host. The failure mode is not cost, it is drift: the file is written once, links move, and nobody notices because nothing in your stack reads it.

That gives the asymmetry that actually decides this. If no operator ever fetches your file, you spent an hour. If one does and yours has drifted, you have handed a machine incorrect information about your own site, with your authority behind it. The second outcome is the one worth engineering against, and it argues for checking the URLs in it on the same schedule you check your sitemap — not for skipping the file.

Why “does it work” is the wrong question #

The phrasing assumes the file is a mechanism, something that acts on a system and produces an outcome. It is not. It is a declaration, and a declaration is judged on whether it is accurate and legible, not on whether anybody has acted on it yet.

Two better questions. Does having written it make your site easier to describe correctly — which it usually does, because the act of choosing twenty pages and summarising each one is an audit of your own information architecture that most sites have never performed. And would you notice if it became wrong — which is a monitoring question with a yes-or-no answer.

Answer those two and the adoption debate stops mattering. You end up with an accurate, current, machine-readable statement of what your site is for, which is worth having whether or not a model provider ever announces it reads one.

What to do #

Publish one if it costs you an hour, keep it accurate, and check that it serves text/plain or text/markdown on the bare URL rather than a cached challenge page. Then stop thinking about it, because it is not the constraint.

The constraint is upstream: 17.4% of scans here serve crawlers a stale cached object, 27.6% deliver a page that is overwhelmingly code, and 5.8% refuse at least one answer engine outright. An llms.txt on a site with any of those is a note pinned to a locked door. Fix the door, publish the note, and treat anyone selling llms.txt as a growth strategy with the scepticism that claim has earned.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• agents.md template inheritance: is the file on your domain yours?
• 33 AI visibility tools measured: a third publish llms.txt, none an entity map

Questions this post answers

Is llms.txt worth it?

It costs about an hour and cannot hurt if kept accurate, but it is not the constraint on whether answer engines can use your site. No model provider publicly documents it as an input, roughly 70,213 sites publish one, and only about a third of 33 AI-visibility vendors publish one on their own domain.

Is llms.txt a standard?

It is a convention, not a ratified standard, and it is separate from robots.txt. It describes content and grants no permissions. Nothing in it can allow or block a crawler.

How would I know if anything reads my llms.txt?

Check your server access logs for requests to the file, and verify each request's source address against the operator's published IP ranges before counting it. Raw log lines naming a crawler are unreliable: 37% of checkable crawler claims in this dataset came from outside the published range.

Does llms.txt improve AI visibility?

There is no public evidence that it does. No major model provider documents it as an input, and a site cannot observe from outside whether anything fetched it. The arguments in its favour are inference: it is cheap, it is a well-formed statement of what you consider important, and it is prose rather than markup. Treat it as a declaration, not as a ranking factor.

Is llms.txt the same as robots.txt?

No. robots.txt sets permissions and points at your sitemap, and it is honoured by operators that choose to honour it. llms.txt is a content map that restricts nothing and grants nothing. Publishing one has no effect on what any crawler is allowed to fetch, and it cannot be used to block anything.

How do I know if my llms.txt is being read?

Only from your own server logs. Filter requests for the file path, group them by source address, and check those addresses against the ranges the named operators publish. That tells you whether it is fetched, by whom, and how often, which is the one question about llms.txt that cannot be answered by scanning a site from outside.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works