CrawlCheck

Datasets · by problem

A sitemap answers 200 and lists nothing

The file parses as a sitemap and contains no URLs. A crawler that follows the declaration fetches it on every visit and learns nothing, and Search Console reports 0 discovered URLs for it. The usual cause is a generator with an empty source — an entity-map, image or news sitemap whose feed returned nothing — or a rewrite rule left behind. Either populate it or stop declaring it. No site in the newest files has it; the list fills as sites with it are measured.

  • 0 sites, last 7 days
  • 0 US local businesses
  • 0% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.