CrawlCheck

Datasets · by problem

The ARD manifest is valid but its entries will not be found by search

The ARD manifest validates, but entries are missing representativeQueries (or carry fewer than 2 or more than 5), use a renamed media type, or carry non-standard parameters. The semantic index a registry builds comes from representativeQueries: an entry without them is a valid catalogue row that search never returns. 188 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 188 sites, measured in the last 7 days
  • 54 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $99 once, about 53¢ a row, 30 days of downloads

At a glance: 67% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is C (55%). How CrawlCheck measures

Who has it

Platform

No CMS detected 67%Next.js 24%WordPress 6%Webflow 2%Shopify 1%

Grade

C 55%A 31%B 9%D 4%F 1%

Where

SaaS and software 47%US local businesses 29%Wider corpus 23%Government and education 1%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherAWordPressblocked
SaaS and softwareGov EduANo CMS detectedallowed
SaaS and softwareOtherANext.jsallowed
PennsylvaniaRestaurantCNo CMS detectedallowed
Wider corpusOtherCNo CMS detectedallowed
SaaS and softwareSaasANo CMS detectedallowed
SaaS and softwareOtherBNo CMS detectedblocked
Hilton, New YorkDentistCNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.