robots.txt answers with the catch-all page - there is no robots.txt
The site answers robots.txt with the same page it serves for a path that does not exist. Crawlers cannot parse HTML as robots rules, so they proceed as if no file existed: everything allowed. Publish a real text/plain robots.txt. 5,187 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 5,187 sites, last 7 days
- 1,079 US local businesses
- 0% also block an AI crawler
- $199 once, 30 days of downloads
Who has it
Platform
No CMS detected 75%WordPress 11%Next.js 8%Drupal 3%Webflow 1%Shopify 1%
Grade
D 74%F 22%C 4%
Where
Wider corpus 47%US local businesses 21%SaaS and software 19%Government and education 13%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | F | Next.js | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.