www and the apex both serve the site instead of one redirecting
Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page. 11,691 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 11,691 sites, measured in the last 7 days
- 3152 US local businesses
- 2026-11-17 last counted, recounted every hour
- $299 once, about 2.6¢ a row, 30 days of downloads
At a glance: 87% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is D (50%). How CrawlCheck measures
Who has it
Platform
No CMS detected 87%Next.js 5%WordPress 3%Drupal 3%Webflow 1%Squarespace 0%
Grade
D 50%C 42%F 8%
Where
Government and education 31%US local businesses 27%SaaS and software 25%Wider corpus 17%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Saas | C | Drupal | allowed |
| SaaS and software | Other | F | WordPress | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | C | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | D | No CMS detected | blocked |
| SaaS and software | Saas | D | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.