CrawlCheck

Datasets · by problem

A child sitemap named in the sitemap index does not load

The sitemap index lists a child sitemap that answers 404, an error or an HTML page. Every page listed only in that child is missing from the sitemap a crawler reads, so new and deep pages there are found late or not at all. 712 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 712 sites, measured in the last 7 days
  • 415 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $99 once, about 14¢ a row, 30 days of downloads

At a glance: 46% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (77%). How CrawlCheck measures

Who has it

Platform

No CMS detected 46%WordPress 43%Next.js 5%Wix 3%Shopify 1%Drupal 1%

Grade

C 77%D 18%F 5%

Where

US local businesses 58%SaaS and software 16%Wider corpus 15%Government and education 11%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherCNo CMS detectedallowed
SaaS and softwareOtherCNext.jsallowed
Government and educationGov EduDWordPressallowed
Whitehall, PennsylvaniaInsuranceCWordPressallowed
North East, PennsylvaniaRestaurantCWordPressallowed
PennsylvaniaOutdoorCWordPressallowed
Harrisville, PennsylvaniaYesCNo CMS detectedallowed
Stroudsburg, PennsylvaniaLawyerCNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.