CrawlCheck

Datasets · by problem

An AI opt-out signal is set

A Content-Signal directive is telling AI systems not to use this site. CDNs inject this by default on some plans, so it is frequently set without the owner deciding to set it. 77 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 77 sites, last 7 days
  • 19 US local businesses
  • 58% also block an AI crawler
  • $99 once, 30 days of downloads

Who has it

Platform

No CMS detected 48%Drupal 31%Next.js 13%WordPress 6%Webflow 1%

Grade

C 55%D 36%F 9%

Where

Wider corpus 32%US local businesses 25%Government and education 25%SaaS and software 18%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherCNext.jsallowed
Camas, WashingtonEstate AgentCNo CMS detectedblocked
Virginia Beach, VirginiaEstate AgentCNo CMS detectedblocked
Government and educationGov EduCDrupalblocked
South Amboy, New JerseyCar RentalFNo CMS detectedallowed
Government and educationGov EduDDrupalblocked
Wider corpusOtherDDrupalblocked
Government and educationGov EduDDrupalblocked

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.