The sitemap is refused
SAME LIMIT AS ABOVE: if a named application firewall appears in the response, this may be blocking CrawlCheck's IP rather than crawlers. Confirm in server logs grouped by source IP. Otherwise: the server refuses this sitemap rather than returning it or a clean 404. Crawlers cannot enumerate the site and fall back to link-following, so deep and newly published pages go undiscovered. 3,604 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 3,604 sites, last 7 days
- 1,827 US local businesses
- 7% also block an AI crawler
- $199 once, 30 days of downloads
Who has it
Platform
No CMS detected 51%Wix 35%WordPress 6%Next.js 4%Drupal 3%Squarespace 0%
Grade
D 94%F 6%
Where
US local businesses 51%Wider corpus 32%Government and education 9%SaaS and software 8%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | D | Next.js | allowed |
| SaaS and software | Saas | D | Next.js | allowed |
| SaaS and software | Other | D | Wix | allowed |
| SaaS and software | Other | D | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.