The sitemap returns HTML, not XML
A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered. 379 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 379 sites, last 7 days
- 96 US local businesses
- 15% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 66%WordPress 27%Next.js 6%Drupal 1%Shopify 0%Squarespace 0%
Grade
F 100%
Where
Wider corpus 55%US local businesses 25%Government and education 11%SaaS and software 9%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Saas | F | Next.js | allowed |
| Government and education | Gov Edu | F | No CMS detected | allowed |
| Arizona | Dentist | F | WordPress | allowed |
| Wider corpus | Local Service | F | No CMS detected | allowed |
| Wider corpus | Other | F | No CMS detected | allowed |
| Government and education | Gov Edu | F | Next.js | allowed |
| Government and education | Gov Edu | F | No CMS detected | allowed |
| Dayton, Washington | Estate Agent | F | WordPress | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.