robots.txt rules name a retired crawler, not the one that visits
The file writes rules for a crawler name its operator no longer lists (anthropic-ai or Claude-Web for Anthropic, FacebookBot for Meta). The crawler that actually visits uses its current name and obeys only a group naming it, so it follows the `*` group instead and the rule you wrote never reaches it. Reported only when that changes what the crawler may fetch. 206 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 206 sites, measured in the last 7 days
- 18 US local businesses
- 2026-11-03 last counted, recounted every hour
- $99 once, about 48¢ a row, 30 days of downloads
At a glance: 67% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (43%). How CrawlCheck measures
Who has it
Platform
No CMS detected 67%Next.js 17%WordPress 13%Drupal 3%
Grade
C 43%D 41%F 16%
Where
Wider corpus 88%US local businesses 9%Government and education 2%SaaS and software 1%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| Government and education | Gov Edu | F | No CMS detected | allowed |
| Iowa | Attraction | D | WordPress | blocked |
| Wider corpus | Saas | C | Next.js | allowed |
| Huntsville, Utah | Resort | D | No CMS detected | blocked |
| Salt Lake City, Utah | Studio | C | WordPress | blocked |
| Wider corpus | Other | D | No CMS detected | blocked |
| Wider corpus | Other | D | No CMS detected | blocked |
| Wider corpus | Other | D | Next.js | blocked |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.