The site refuses plain automated requests; it was measured through a real browser
The site’s security layer refused CrawlCheck’s plain HTTP request but served the page to a real browser. Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) make plain HTTP requests and do not run JavaScript, so a shield that tells the two apart this way is likely refusing them too, which keeps the site out of AI answers no matter what robots.txt allows. The rest of this report was measured through the browser, so its findings describe the site, not the wall. 25 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 25 sites, last 7 days
- 0 US local businesses
- 16% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 60%WordPress 20%Next.js 12%Wix 4%Drupal 4%
Grade
D 44%C 28%F 16%B 12%
Where
Wider corpus 100%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| Wider corpus | Local Service | C | Wix | allowed |
| Wider corpus | Other | C | Drupal | allowed |
| Wider corpus | Other | D | No CMS detected | allowed |
| Wider corpus | Other | B | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.