CrawlCheck

Glossary · Machine files

sitemap index

A sitemap that lists other sitemaps rather than pages. Counting its entries as URLs undercounts a large site by orders of magnitude.

Terms this definition uses

Sitemap

Machine files

The files addressed to machines rather than readers, and what publishing one does and does not buy.

llms.txt · llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · TDM exception · security.txt

agent card  ·  OpenAPI specification

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.