A named crawler was served materially less text than an ordinary client
Same URL, same minute, both answered 200 — but the named crawler received materially fewer words than an unnamed client did. A refusal is loud and easy to find; being served a thinner page is silent, and it is the version an answer engine quotes from. No site in the newest files has it; the list fills as sites with it are measured.
- 0 sites, last 7 days
- 0 US local businesses
- 0% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.