CrawlCheck

Datasets · by problem

The homepage is reached through a chain of redirects

Reaching the homepage takes more than one redirect, for example http to https and then the domain to www. Each hop is another request and round trip before a crawler or visitor gets the page, temporary hops (302, 307) tell search engines to keep the old address, and Google stops following after ten. 44,997 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 44,997 sites, last 7 days
  • 17,646 US local businesses
  • 4% also block an AI crawler
  • $299 once, 30 days of downloads

Who has it

Platform

No CMS detected 46%WordPress 30%Wix 8%Next.js 8%Drupal 2%Webflow 2%

Grade

C 35%D 29%A 18%B 12%F 7%

Where

Wider corpus 40%US local businesses 39%SaaS and software 12%Government and education 8%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareLocal ServiceCWixallowed
SaaS and softwareOtherCNext.jsallowed
SaaS and softwareOtherANo CMS detectedallowed
SaaS and softwareOtherCWixallowed
SaaS and softwareSaasANo CMS detectedallowed
SaaS and softwarePublisherDNo CMS detectedallowed
SaaS and softwareOtherANo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.