machine layer
Everything a site serves to non-human clients: robots.txt, sitemaps, llms.txt, agents.md, entity maps, structured data and response headers.
What machine layer means
A site can look finished to a visitor and be nearly empty at this layer.
How CrawlCheck measures machine layer
2 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
MACHINE_CHAIN_HEAVY | 22% | Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits |
NO_LLMS_TXT | 32.5% | Does llms.txt work? What is known and how to test it |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where machine layer comes up on this site
- How to measure AI visibility yourself: three checksIf you are absent from answers and your machine layer is broken, you have learned nothing about your content or your market: you have measured a cache.
- AI visibility vendors audited: 11 of 23 have llms.txtA vendor who fixes their machine layer tomorrow deserves the better number, and the directory ledger records the change with its date.
Terms this definition uses
robots.txt · Sitemap · llms.txt · agents.md · entity map · structured data
Definitions that name machine layer
Related terms in Machine files
The files addressed to machines rather than readers, and what publishing one does and does not buy.
llms.txt · llms-full.txt · agents.md · entity map · entity map parity · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · TDM exception · security.txt · Content Signals · HTTP Message Signatures · changefreq and priority · staging leak · host duplicate
Questions about machine layer
What is machine layer?
Everything a site serves to non-human clients: robots.txt, sitemaps, llms.txt, agents.md, entity maps, structured data and response headers.
How does CrawlCheck measure machine layer?
Through the finding codes MACHINE_CHAIN_HEAVY, NO_LLMS_TXT, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does machine layer belong to?
Machine files: The files addressed to machines rather than readers, and what publishing one does and does not buy.
← entity map parity · well-known URI →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.