CrawlCheck

Datasets · by problem

No XML sitemap found

Without a sitemap, discovery depends entirely on internal linking. Orphaned and recently published pages may never be found. 28,053 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 28,053 sites, last 7 days
  • 5,331 US local businesses
  • 4% also block an AI crawler
  • $299 once, 30 days of downloads

Who has it

Platform

No CMS detected 82%WordPress 7%Next.js 6%Drupal 3%Webflow 1%Squarespace 0%

Grade

D 83%F 17%

Where

Wider corpus 51%US local businesses 19%SaaS and software 18%Government and education 13%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareOtherDNext.jsallowed
SaaS and softwareSaasDNext.jsallowed
SaaS and softwareOtherDNext.jsallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareSaasDNo CMS detectedallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareOtherDNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.