CrawlCheck

Datasets · by problem

The homepage answered with an interstitial, not a page

133 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 133 sites, last 7 days
  • 55 US local businesses
  • 0% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

Platform

No CMS detected 100%

Where

US local businesses 41%Wider corpus 39%SaaS and software 12%Government and education 8%

WhereCategoryGradePlatformAI crawlers
Government and educationGov Edu–No CMS detectedallowed
Flagstaff, ArizonaEstate Agent–No CMS detectedallowed
Edmonds, WashingtonDentist–No CMS detectedallowed
Seattle, WashingtonVeterinary–No CMS detectedallowed
Wider corpusOther–No CMS detectedallowed
Wider corpusOther–No CMS detectedallowed
Wider corpusOther–No CMS detectedallowed
Hammonton, New JerseyVeterinary–No CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.