CrawlCheck

Datasets · by problem

www and the apex both serve the site instead of one redirecting

Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page. 11,691 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 11,691 sites, measured in the last 7 days
  • 3152 US local businesses
  • 2026-11-17 last counted, recounted every hour
  • $299 once, about 2.6¢ a row, 30 days of downloads

At a glance: 87% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is D (50%). How CrawlCheck measures

Who has it

Platform

No CMS detected 87%Next.js 5%WordPress 3%Drupal 3%Webflow 1%Squarespace 0%

Grade

D 50%C 42%F 8%

Where

Government and education 31%US local businesses 27%SaaS and software 25%Wider corpus 17%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareSaasCDrupalallowed
SaaS and softwareOtherFWordPressallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareSaasCNo CMS detectedallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareSaasDNo CMS detectedblocked
SaaS and softwareSaasDNext.jsallowed
SaaS and softwareOtherDNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.