CrawlCheck

Datasets · by problem

The page's date changed but its words did not

Between two of CrawlCheck's reads, the page's declared modification date (dateModified or article:modified_time) moved forward while the main content, read with navigation, ads and dates removed, stayed the same. A date that moves without new words claims freshness the page does not have; engines that compare the two learn to ignore the date. No site in the newest files has it; the list fills as sites with it are measured.

  • 0 sites, last 7 days
  • 0 US local businesses
  • 0% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.