Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
How to write an llms.txt, and how to check it is actually being served
The file takes twenty minutes. Confirming that the version a crawler receives is the version you wrote takes two more, and it is the step almost everyone skips.
An llms.txt is a markdown file at the root of your domain describing what the site contains and where the important pages are. It is a content map. It grants no permissions and blocks nothing — that is robots.txt's job — and no model provider publicly documents it as a required input. Roughly 70,213 sites publish one.
It is cheap, and cheap is the whole case for it. What follows is how to do it in a way that costs an hour rather than becoming another unmaintained artefact on your domain.
The structure
The convention is plain markdown with a specific shape:
| Element | Purpose |
|---|---|
# Site name | One H1, the name of the entity |
| > blockquote | One or two sentences on what the site is |
| Free prose | Optional context: what you do, where, for whom |
## Section headings | Groupings — Docs, Services, About, Pricing |
| Link lines | - [Title](url): description |
## Optional | Pages that may be skipped under a budget |
The descriptions are the part that carries the value, and they are the part most files omit. A list of forty bare URLs with no descriptions is a sitemap in worse clothing. The point of the format is that a reader can decide what is worth fetching without fetching it — which requires you to have said what each thing is.
What to include, in priority order
Start from the questions people actually ask about you and work backwards. Web-wide, Organization is detected on roughly 354,000 sites and OpeningHoursSpecification on roughly 23,000: declaring you exist is common, declaring the facts anyone asked about is an order of magnitude rarer.
So: the pages that state what you sell and what it costs. The page that states where you are and when you are open. The pages that answer the five questions your customers ask before buying. Your documentation entry points, if you have docs. Then stop. A twenty-line file that is accurate beats a two-hundred-line file that drifts.
Are generators worth it?
For a first draft, they save time, and there is nothing wrong with that. Two cautions.
Generated files inherit their generator's assumptions. Most produce a link dump from your sitemap with titles as descriptions, which reproduces exactly the failure above — no editorial judgement about what matters, because the tool has none to offer. The value of the file is the choosing, and the choosing is the part a generator cannot do.
A generated file is a snapshot. It was true on the day it ran. Nothing regenerates it when you launch a service line or retire a page, so unless you wire it into a build step, it decays quietly from the day it ships.
The pattern that works: generate, then delete two-thirds of it, then write the descriptions yourself.
Verify it is served — the step that gets skipped
Publishing is not the same as serving. Check the bare URL, with no cache-buster, because a crawler never appends one:
curl -sI https://yoursite.com/llms.txt
Two things must be true. The status is 200, and the content type is text/plain or text/markdown. If it is text/html, something is answering in place of your file — commonly a bot-verification interstitial, which is read as the file itself by anything that does not run JavaScript.
Then confirm the body is yours:
curl -s https://yoursite.com/llms.txt | head -20
Machine files go stale in transit more often than people expect: MACHINE_FILE_CACHE_STALE fires on 1.6% of scans and MACHINE_FILE_CACHE_SPLIT on 0.2% — cases where what a crawler receives is not what the origin holds. Across all machine files, STALE_CACHE_SERVED fires on 30.9% of scans. A file is not published until the bare URL returns it.
Then declare it
Name it in robots.txt alongside your sitemap so it is discoverable by anything that starts where crawlers start. And check the sitemap declaration while you are in there — ROBOTS_NO_SITEMAP fires on 3.2% of scans, DECLARED_SITEMAP_BROKEN on 2.3%, and SITEMAP_BLOCKED on 1.8%, that last one being a site that declares a sitemap and then disallows the path to it.
The neighbouring file that is not yours
While you are at the root of the domain, check for agents.md. It is a different file with a different job — llms.txt is a content map, agents.md is an instruction file written to be obeyed by an agent — and on several major platforms it ships by default from a shared template.
That means whole hosting populations serve identical instruction text, sometimes including steering language directed at assistants, on domains whose owners have never opened the file. If something on your domain is written to instruct a machine, it is worth knowing whether you wrote it.
curl -s https://yoursite.com/agents.md | head -30
A 200 that returns HTML there is a soft 404 rather than a policy file, and absence is not a defect — adoption is early. What matters is that you know which of the two cases you are in.
Keep expectations calibrated
Only about one in three of 33 AI-visibility vendors scanned publish an llms.txt on their own domain, and none publishes an entity map. The people closest to this advice are not betting heavily on it, and neither should you.
Write it because it is cheap and forces a useful decision. Do not write it expecting it to move anything on its own, and never write it before the site underneath it can be fetched, parsed and quoted.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
What should go in an llms.txt file?
An H1 with the site name, a one-line summary in a blockquote, then sections of link lines in the form of title, URL and a short description. Prioritise the pages stating what you sell, what it costs, where you are and when you are open. Keep it short enough to stay accurate.
Should I use an llms.txt generator?
For a first draft, yes. Then cut most of it and write the descriptions yourself, because a generator produces a link dump from your sitemap with no editorial judgement about what matters, and the choosing is the part that carries the value.
How do I check my llms.txt is working?
Fetch the bare URL with no cache-buster and confirm the status is 200 and the content type is text/plain or text/markdown, then read the first lines to confirm the body is yours. Machine files are served stale or replaced by challenge pages more often than people expect.