CrawlCheck

Datasets · by problem

robots.txt declares a sitemap with a relative path

A Sitemap line in robots.txt gives a path (/sitemap.xml) instead of a full URL. The standard requires a full URL, and crawlers drop a relative one, so the sitemap is not declared to them through robots.txt. 1,129 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 1,129 sites, measured in the last 7 days
  • 384 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $199 once, about 18¢ a row, 30 days of downloads

At a glance: 92% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (76%). How CrawlCheck measures

Who has it

Platform

No CMS detected 92%WordPress 5%Next.js 2%Drupal 1%Webflow 0%Shopify 0%

Grade

C 76%A 10%D 9%B 4%F 1%

Where

Government and education 51%US local businesses 34%Wider corpus 9%SaaS and software 6%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareEcommerceBNo CMS detectedallowed
SaaS and softwareOtherCNo CMS detectedallowed
SaaS and softwareOtherCNo CMS detectedallowed
Government and educationGov EduCNo CMS detectedallowed
Government and educationGov EduCNo CMS detectedallowed
Government and educationGov EduCNo CMS detectedallowed
Government and educationGov EduCNo CMS detectedallowed
Government and educationGov EduCWordPressallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.