CrawlCheck

Datasets · by problem

The homepage sends crawlers to a different hostname

A request for this domain's homepage answers with a redirect to a hostname outside its own www/apex pair - a regional storefront, a checkout host, a vendor platform. CrawlCheck does not follow it, and neither does a crawler attribute what it finds there to this domain: content, schema and entity signals on the other host belong to the other host, and this domain reads as a site with no homepage. If the redirect is deliberate, the domain still needs a page of its own that a crawler can read. No site in the newest files has it; the list fills as sites with it are measured.

  • 0 sites, last 7 days
  • 0 US local businesses
  • 0% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.