A security layer refused this file — source unconfirmed
The cached copy is fine; behind it the origin refused this request. IMPORTANT LIMIT: a scanner runs from one IP, and application firewalls block by IP reputation — so this may mean the site refuses crawlers, or it may only mean the site refuses CrawlCheck specifically. The two are indistinguishable from outside. Confirm in the server access logs, grouped by source IP, before acting on it. 74 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 74 sites, last 7 days
- 12 US local businesses
- 16% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 64%WordPress 36%
Grade
D 74%C 26%
Where
Wider corpus 73%US local businesses 16%SaaS and software 7%Government and education 4%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Saas | C | No CMS detected | blocked |
| Marysville, Washington | Dentist | D | WordPress | allowed |
| Wider corpus | Other | D | WordPress | blocked |
| Bellmawr, New Jersey | Place Of Worship | D | WordPress | blocked |
| Wider corpus | Other | D | No CMS detected | allowed |
| Wider corpus | Other | D | No CMS detected | allowed |
| Wider corpus | Other | D | No CMS detected | allowed |
| Wider corpus | Other | D | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.