llms.txt
A Markdown file at the root of a site that tells language models what the site is and which pages matter.
What llms.txt does not show
It is a proposal, not a standard: no engine has committed to reading it, and publishing one is a low-cost bet rather than a ranking action.
How CrawlCheck measures llms.txt
2 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
NO_LLMS_TXT | 32.5% | Does llms.txt work? What is known and how to test it |
LLMS_LINK_BROKEN | 0.4% | dataset |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where llms.txt comes up on this site
- Does llms.txt work? What is known and how to test itllms.txt is a Markdown file at the root of a domain that says what a site contains and where the important pages are.
- How to write an llms.txt and check it is actually servedAn llms.txt is a Markdown file at the root of your domain that says what the site contains and where the important pages are, with one line describing each.
- Stale cached robots.txt or llms.txt? Two fetches tell yourobots.txt, llms.txt, sitemap.xml and an entity map are small files that rarely change, which is exactly what a CDN caches hardest.
- Best llms.txt generators and validators comparedAn llms.txt is a Markdown file at the root of a site that points AI tools to its most useful pages.
- AI visibility vendors audited: 11 of 23 have llms.txtDisclosure: CrawlCheck publishes this piece and sells a tool in the same category. crawlcheck.io is not in its own corpus, because a scanner cannot grade its own host from the outside view it uses for everyone else; check our files from any client you…
Definitions that name llms.txt
fetcher disagreement · soft 404 on a machine file · machine layer · llms-full.txt · OpenAPI specification · ai.txt · security.txt · signed absence · variableMeasured
Related terms in Machine files
The files addressed to machines rather than readers, and what publishing one does and does not buy.
llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · TDM exception · security.txt · Content Signals · HTTP Message Signatures · changefreq and priority · staging leak · host duplicate
Questions about llms.txt
What is llms.txt?
A Markdown file at the root of a site that tells language models what the site is and which pages matter.
What does llms.txt not show or guarantee?
It is a proposal, not a standard: no engine has committed to reading it, and publishing one is a low-cost bet rather than a ranking action.
How does CrawlCheck measure llms.txt?
Through the finding codes NO_LLMS_TXT, LLMS_LINK_BROKEN, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does llms.txt belong to?
Machine files: The files addressed to machines rather than readers, and what publishing one does and does not buy.
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.