soft 404 on a machine file
A platform that answers every unknown path with its homepage answers /llms.txt and /sitemap.xml that way too, at HTTP 200.
What soft 404 on a machine file does not show
A nonexistent-path control fetched in the same scan identifies the catch-all page, and any machine file that matches it is recorded as absent rather than present or broken.
How CrawlCheck measures soft 404 on a machine file
2 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
SITEMAP_IS_HTML | 0.4% | Do AI crawlers and agents respect robots.txt? |
ROBOTS_IS_HTML | 0.1% | How to check if AI crawlers can read your site |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Terms this definition uses
llms.txt · Sitemap · nonexistent-path control
Related terms in Machine files
The files addressed to machines rather than readers, and what publishing one does and does not buy.
llms.txt · llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · tdmrep.json · TDM exception · security.txt · Content Signals · HTTP Message Signatures · changefreq and priority · staging leak · host duplicate
Questions about soft 404 on a machine file
What is soft 404 on a machine file?
A platform that answers every unknown path with its homepage answers /llms.txt and /sitemap.xml that way too, at HTTP 200.
What does soft 404 on a machine file not show or guarantee?
A nonexistent-path control fetched in the same scan identifies the catch-all page, and any machine file that matches it is recorded as absent rather than present or broken.
How does CrawlCheck measure soft 404 on a machine file?
Through the finding codes SITEMAP_IS_HTML, ROBOTS_IS_HTML, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does soft 404 on a machine file belong to?
Machine files: The files addressed to machines rather than readers, and what publishing one does and does not buy.
← sitemap origin error · tdmrep.json →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.