CrawlCheck

Datasets · by problem

The sitemap lists URLs on another host

The sitemap names URLs on a different host from the sitemap itself. The sitemap protocol allows only URLs on its own host, so crawlers drop them; if the other host is a staging or old domain, it also advertises that copy. 5,450 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 5,450 sites, measured in the last 7 days
  • 1906 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $199 once, about 3.7¢ a row, 30 days of downloads

At a glance: 48% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (51%). How CrawlCheck measures

Who has it

Platform

No CMS detected 48%WordPress 27%Next.js 9%Drupal 5%Squarespace 4%Wix 3%

Grade

C 51%A 26%B 13%D 6%F 4%

Where

US local businesses 35%SaaS and software 27%Wider corpus 24%Government and education 14%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherCWordPressallowed
SaaS and softwareOtherANo CMS detectedallowed
SaaS and softwareDocsANo CMS detectedallowed
SaaS and softwareSaasANo CMS detectedblocked
SaaS and softwareLocal ServiceANo CMS detectedallowed
SaaS and softwareSaasANext.jsallowed
SaaS and softwareSaasBNo CMS detectedallowed
SaaS and softwareSaasCNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.