CrawlCheck

Glossary · Machine files

soft 404 on a machine file

A platform that answers every unknown path with its homepage answers /llms.txt and /sitemap.xml that way too, at HTTP 200. A nonexistent-path control fetched in the same scan identifies the catch-all page, and any machine file that matches it is recorded as absent rather than present or broken.

Terms this definition uses

llms.txt · Sitemap · nonexistent-path control

Machine files

The files addressed to machines rather than readers, and what publishing one does and does not buy.

llms.txt · llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · tdmrep.json · TDM exception · security.txt

sitemap origin error  ·  tdmrep.json

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.