CrawlCheck

Glossary · Machine files

llms.txt

A plain-text file at the root of a site that tells language models what the site is and which pages matter. It is a proposal, not a standard: no engine has committed to reading it, and publishing one is a low-cost bet rather than a ranking action.

Measured live on /data.

Machine files

The files addressed to machines rather than readers, and what publishing one does and does not buy.

llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · TDM exception · security.txt

llms-full.txt

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.