CrawlCheck

Datasets · by problem

robots.txt has a group for a misspelt crawler name

A group is written for a name no crawler sends, one spelling away from a real one (Perplexity-Bot for PerplexityBot, GPT-Bot for GPTBot). A crawler obeys only a group naming its own token, so the real crawler follows the `*` group instead and the rules you wrote for it never reach it. Reported only when that changes what the crawler may fetch. 1,665 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 1,665 sites, measured in the last 7 days
  • 294 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $199 once, about 12¢ a row, 30 days of downloads

At a glance: 87% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (65%). How CrawlCheck measures

Who has it

Platform

No CMS detected 87%WordPress 9%Next.js 2%Squarespace 1%Drupal 0%Webflow 0%

Grade

C 65%D 34%F 2%

Where

Wider corpus 67%US local businesses 18%SaaS and software 12%Government and education 4%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherCNo CMS detectedallowed
SaaS and softwareOtherCNo CMS detectedallowed
SaaS and softwareOtherDNo CMS detectedallowed
Government and educationGov EduCNo CMS detectedblocked
Government and educationGov EduCNo CMS detectedallowed
PennsylvaniaPhotographerCNo CMS detectedallowed
Pittsburgh, PennsylvaniaNgoCWordPressallowed
PennsylvaniaSports CentreDNo CMS detectedblocked

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.