CrawlCheck

Datasets · by problem

This file answers differently depending on whether the cache is hit

38 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 38 sites, last 7 days
  • 8 US local businesses
  • 5% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

Platform

WordPress 68%No CMS detected 29%Next.js 3%

Grade

D 71%F 29%

Where

Wider corpus 61%US local businesses 21%Government and education 13%SaaS and software 5%

WhereCategoryGradePlatformAI crawlers
Government and educationGov EduDWordPressallowed
Wider corpusOtherFNo CMS detectedblocked
New JerseyNature ReserveDWordPressallowed
Rocky Mount, North CarolinaCar RepairDNo CMS detectedblocked
Wider corpusOtherDWordPressallowed
Wider corpusOtherDNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.