robots.txt is unreadable
A 5xx on robots.txt tells a standards-following crawler to treat the whole site as disallowed (RFC 9309). A 401 or 403 is a 4xx, which the standard treats as no file and so allows everything, though some crawlers read it as a refusal. Either way the rules you wrote are not the rules a crawler read. This is the single most expensive file on the site to get wrong. 557 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 557 sites, last 7 days
- 104 US local businesses
- 0% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 74%WordPress 16%Next.js 6%Drupal 3%Squarespace 1%Webflow 0%
Grade
D 85%F 15%
Where
Wider corpus 49%US local businesses 19%SaaS and software 17%Government and education 15%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| Government and education | Gov Edu | F | No CMS detected | allowed |
| Wider corpus | Saas | D | Next.js | allowed |
| Wider corpus | Publisher | D | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.