Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 0 views

How to write an llms.txt, and how to check it is actually being served

The file takes twenty minutes. Confirming that the version a crawler receives is the version you wrote takes two more, and it is the step almost everyone skips.

An llms.txt is a markdown file at the root of your domain describing what the site contains and where the important pages are. It is a content map. It grants no permissions and blocks nothing — that is robots.txt's job — and no model provider publicly documents it as a required input. Roughly 70,213 sites publish one.

It is cheap, and cheap is the whole case for it. What follows is how to do it in a way that costs an hour rather than becoming another unmaintained artefact on your domain.

The structure #

The convention is plain markdown with a specific shape:

The descriptions are the part that carries the value, and they are the part most files omit. A list of forty bare URLs with no descriptions is a sitemap in worse clothing. The point of the format is that a reader can decide what is worth fetching without fetching it — which requires you to have said what each thing is.

What to include, in priority order #

Start from the questions people actually ask about you and work backwards. Web-wide, Organization is detected on roughly 354,000 sites and OpeningHoursSpecification on roughly 23,000: declaring you exist is common, declaring the facts anyone asked about is an order of magnitude rarer.

So: the pages that state what you sell and what it costs. The page that states where you are and when you are open. The pages that answer the five questions your customers ask before buying. Your documentation entry points, if you have docs. Then stop. A twenty-line file that is accurate beats a two-hundred-line file that drifts.

Are generators worth it? #

For a first draft, they save time, and there is nothing wrong with that. Two cautions.

Generated files inherit their generator's assumptions. Most produce a link dump from your sitemap with titles as descriptions, which reproduces exactly the failure above — no editorial judgement about what matters, because the tool has none to offer. The value of the file is the choosing, and the choosing is the part a generator cannot do.

A generated file is a snapshot. It was true on the day it ran. Nothing regenerates it when you launch a service line or retire a page, so unless you wire it into a build step, it decays quietly from the day it ships.

The pattern that works: generate, then delete two-thirds of it, then write the descriptions yourself.

Verify it is served — the step that gets skipped #

Publishing is not the same as serving. Check the bare URL, with no cache-buster, because a crawler never appends one:

curl -sI https://yoursite.com/llms.txt

Two things must be true. The status is 200, and the content type is text/plain or text/markdown. If it is text/html, something is answering in place of your file — commonly a bot-verification interstitial, which is read as the file itself by anything that does not run JavaScript.

Then confirm the body is yours:

curl -s https://yoursite.com/llms.txt | head -20

Machine files go stale in transit more often than people expect: MACHINE_FILE_CACHE_STALE fires on 0.5% of scans and MACHINE_FILE_CACHE_SPLIT on 0.1% — cases where what a crawler receives is not what the origin holds. Across all machine files, STALE_CACHE_SERVED fires on 17.4% of scans. A file is not published until the bare URL returns it.

Then declare it #

Name it in robots.txt alongside your sitemap so it is discoverable by anything that starts where crawlers start. And check the sitemap declaration while you are in there — ROBOTS_NO_SITEMAP fires on 8.8% of scans, DECLARED_SITEMAP_BROKEN on 1.2%, and SITEMAP_BLOCKED on 2.2%, that last one being a site that declares a sitemap and then disallows the path to it.

The neighbouring file that is not yours #

While you are at the root of the domain, check for agents.md. It is a different file with a different job — llms.txt is a content map, agents.md is an instruction file written to be obeyed by an agent — and on several major platforms it ships by default from a shared template.

That means whole hosting populations serve identical instruction text, sometimes including steering language directed at assistants, on domains whose owners have never opened the file. If something on your domain is written to instruct a machine, it is worth knowing whether you wrote it.

curl -s https://yoursite.com/agents.md | head -30

A 200 that returns HTML there is a soft 404 rather than a policy file, and absence is not a defect — adoption is early. What matters is that you know which of the two cases you are in.

Keep expectations calibrated #

Only about one in three of 33 AI-visibility vendors scanned publish an llms.txt on their own domain, and none publishes an entity map. The people closest to this advice are not betting heavily on it, and neither should you.

Write it because it is cheap and forces a useful decision. Do not write it expecting it to move anything on its own, and never write it before the site underneath it can be fetched, parsed and quoted.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• agents.md template inheritance: is the file on your domain yours?
• Your new cache rule did not fix the object it was written for

The three files are not interchangeable, and conflating them is the common mistake #

Most confusion about llms.txt comes from treating it as a permission file. It is not one. Three files sit on a domain and each answers a different question, which is why a site can be perfectly configured in one and broken in another.

The ordering matters. A disallowed path in robots.txt is not rescued by listing it in llms.txt: a crawler that honours the first will never reach the second. Check the permission layer before you spend time on the pointer layer.

The five ways a valid file is served badly #

Writing the file is the easy half. Every failure below produces a file that looks correct in an editor and is useless on the wire, and all five are things a fetch can settle in seconds.

Served as HTML. A framework that renders a catch-all route will answer /llms.txt with your 404 page and status 200. Nothing about that response says “missing”; it is a soft 404, and a parser reads it as a file whose first line is <!DOCTYPE html>. The same failure on robots.txt has its own code, ROBOTS_IS_HTML, because it is common enough to need one.

Wrong content type. text/html on a .txt path is not fatal to every client, but it is a signal that the route is being handled by your application rather than served as a file, which is usually the cause of the previous problem rather than a separate one.

Only on one host. Apex and www are different hosts. A file deployed to one and not the other is present for half the requests that will ever ask for it. CrawlCheck checks machine files on both hosts and reports when they disagree.

Cached stale. A CDN that cached the 404 from before the file existed will keep serving it after you publish. MACHINE_FILE_CACHE_STALE is the code; the fix is a purge, and the thing to check afterwards is the age header rather than your own browser, which has its own cache and will lie to you.

Cached differently per host. One host fresh, one host stale, which is MACHINE_FILE_CACHE_SPLIT and the reason the two-host fetch is worth doing even when you are sure you deployed to both.

Every URL you cite has to resolve, including the redirects #

A pointer file is a list of promises. If a cited URL 404s, or 301s somewhere else, you have spent the attention you were given on a detour. Citing a redirect is the specific case worth naming: it still “works” in a browser, so it survives review, and it is the sloppiness CrawlCheck reports on other people’s machine files constantly.

The check is mechanical. Fetch every URL in the file, without following redirects, and require a 200. Anything else gets fixed or removed before you publish. If you are generating the file, generate it from a source that already knows the canonical URL of each page — your sitemap, not your navigation.

What the corpus says about publishing one at all #

Sites measured here are missing llms.txt at a rate of 31.7%, which makes it the most common absence in the whole scan. That is a fact about adoption, not about value, and the two get confused constantly.

The honest position, stated here because this guide exists to be useful rather than to sell a file: a study across roughly 300,000 domains found no correlation between publishing an llms.txt and being cited by an answer engine. Removing the variable made the model more accurate. CrawlCheck stopped grading the file for the same reason, and reports it without scoring it. If a vendor is selling you an llms.txt as an AI-visibility lever, ask them for the measurement behind the claim.

So why publish one? Because it is cheap, some agents do fetch it, and a malformed one is worth catching. Those are small reasons and they are the real ones. The large reasons — whether a crawler can reach your pages at all, whether what it receives is readable, whether your facts are consistent enough to quote — live in the other two layers, and they are where the time goes if you only have an afternoon.

A twenty-minute version, in order #

If you only have one sitting, do these in this order, because each one makes the next one meaningful.

First, confirm a crawler can reach the pages at all. Read your robots.txt and check that the rules you think apply to named AI crawlers actually apply to them. A group written for one user-agent voids the wildcard rules for that agent entirely, which is how sites end up allowing a crawler they meant to block and blocking one they meant to allow. This is the layer where a mistake costs you everything downstream.

Second, write the file from your sitemap, not your navigation. Your navigation reflects what you want humans to click. Your sitemap reflects the canonical address of each page. Generating from the wrong source is how redirects and duplicate hosts get cited.

Third, publish and purge. Not one or the other. A file that exists at origin and a stale 404 at the edge is indistinguishable, to every client that matters, from a file that was never written.

Fourth, fetch it from outside your own network, on both the apex and the www host, and read the status, the content type and the first line of the body. Your browser is the worst instrument available for this: it has its own cache, it sends your session, and it will render an HTML error page so convincingly that you will believe the file is fine.

Fifth, re-check in a week. Machine files rot quietly. A deploy that changes routing, a CDN rule someone adds, a framework upgrade that starts handling unknown paths — none of these announce themselves, and all of them have taken a working file offline on sites measured here.

Questions this post answers

What should go in an llms.txt file?

An H1 with the site name, a one-line summary in a blockquote, then sections of link lines in the form of title, URL and a short description. Prioritise the pages stating what you sell, what it costs, where you are and when you are open. Keep it short enough to stay accurate.

Should I use an llms.txt generator?

For a first draft, yes. Then cut most of it and write the descriptions yourself, because a generator produces a link dump from your sitemap with no editorial judgement about what matters, and the choosing is the part that carries the value.

How do I check my llms.txt is working?

Fetch the bare URL with no cache-buster and confirm the status is 200 and the content type is text/plain or text/markdown, then read the first lines to confirm the body is yours. Machine files are served stale or replaced by challenge pages more often than people expect.

What should go in an llms.txt file?

A single H1 with the site name, a blockquote summarising what the site is, then linked sections pointing at the pages you most want read, each link followed by a short description of what it contains. Keep it to the pages that answer questions about you. It is an index, not a sitemap, and a long one is worse than a short one.

Where do I put llms.txt and how is it served?

At the root of the domain, alongside robots.txt, returning 200 as Markdown or plain text. The failures worth checking for are an HTML error page returned with a 200 status, a content type of text/html, a redirect to a different host, and a cached copy that no longer matches the origin. Each of those makes the file unreadable to the clients it was written for.

Does llms.txt do anything yet?

No major operator has committed to reading it, so treat it as a cheap, low-risk declaration rather than as a ranking input. The reason to write one anyway is that the work it forces you to do, naming your most useful pages and confirming every one of them resolves, is worth doing regardless of whether any client ever fetches the file.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works