The instruction files contradict each other
A crawler obeys only its own User-agent group. A Content-Signal or Disallow written under * says nothing to an agent that has its own group, and a group that both invites and forbids leaves the crawler to pick. Put the signal in every group it is meant for, and let the rules under it agree. 377 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 377 sites, last 7 days
- 102 US local businesses
- 73% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 60%Next.js 21%WordPress 15%Shopify 2%Webflow 1%Drupal 1%
Grade
C 51%F 20%B 10%A 9%D 9%
Where
Wider corpus 45%US local businesses 27%SaaS and software 25%Government and education 3%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Saas | F | No CMS detected | allowed |
| SaaS and software | Saas | C | Next.js | blocked |
| SaaS and software | Saas | A | Next.js | blocked |
| SaaS and software | Saas | A | No CMS detected | blocked |
| SaaS and software | Other | B | No CMS detected | blocked |
| SaaS and software | Other | A | No CMS detected | allowed |
| Flagstaff, Arizona | Car Repair | C | No CMS detected | blocked |
| Mesa, Arizona | Car Repair | C | No CMS detected | blocked |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.