Some listings can only be reached by filling in a form
Crawlers do not fill in forms. A results page reached only by picking options in a filter form (a location, a category, a price band) and pressing submit is part of the “deep web”: the listings exist for people but no search engine or AI crawler can find them unless a plain link or the sitemap leads there. A lone site-search box is not counted, because search results are meant to stay out of indexes. Reported only when no plain link to the results page exists on the pages read. 29 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 29 sites, last 7 days
- 2 US local businesses
- 3% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
Platform
No CMS detected 76%Drupal 17%Webflow 7%
Grade
D 48%C 48%A 3%
Where
Government and education 38%Wider corpus 34%SaaS and software 21%US local businesses 7%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| Government and education | Gov Edu | D | No CMS detected | allowed |
| Maryland | Camp Site | D | No CMS detected | allowed |
| Wider corpus | Other | D | No CMS detected | allowed |
| SaaS and software | Publisher | C | WordPress | allowed |
| Wider corpus | Other | C | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.