CrawlCheck

Datasets · by problem

The origin returned a server error on the sitemap path

625 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 625 sites, last 7 days
  • 133 US local businesses
  • 4% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

Platform

No CMS detected 72%Next.js 15%WordPress 10%Drupal 1%Wix 0%Webflow 0%

Grade

C 48%D 26%A 11%F 10%B 5%

Where

Wider corpus 53%US local businesses 21%SaaS and software 17%Government and education 8%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherCNext.jsallowed
SaaS and softwareDocsCNo CMS detectedallowed
SaaS and softwareOtherDNext.jsallowed
SaaS and softwareSaasANext.jsallowed
SaaS and softwareSaasCNext.jsallowed
Sedona, ArizonaCar RepairDNo CMS detectedallowed
Scottsdale, ArizonaTheatreDNo CMS detectedallowed
Wider corpusPublisherBNo CMS detectedblocked

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.