Findings · 2026-08-13
We measured what answer engines receive from local business sites. Most of it is not text.
Three findings from a corpus of sites measured byte by byte, and why none of them appear in a validator.
Bars are drawn to scale. The rest of every bar is code the crawler downloads and cannot use.
Every SEO tool measures what a page says. Almost none measure what a machine is handed when it asks for that page. Those are different numbers, and the gap between them is where most of the damage lives.
1. The heaviest page we measured delivered 2.3% text
476,540 bytes on the wire. Visible, readable text: 2.3% of it. The rest was markup, inline styles and scripts. A crawler pays that cost on every URL it fetches, and inline CSS cannot be cached between pages, so it pays again on the next one.
The page looked fine to every human who visited it. It renders quickly, it reads well, and no validator has an opinion about it.
2. A file returning HTTP 200 that no crawler could read
On one site, /robots.txt answered every request with a browser-verification interstitial and a 200 OK. Crawlers do not execute the JavaScript that clears such a page — they read the interstitial as the file. Every uptime monitor called the site healthy.
The cause was not the security product. It was a cache rule storing whatever came back, including a challenge page, and serving it to everyone for the next five minutes. A per-address defence became a site-wide outage of the machine layer.
3. Two engines refused at the edge while Google walked in
On another site, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally. Nothing in robots.txt said so — this was the edge, and almost nobody configures it deliberately.
Until that is lifted, nothing else about the site reaches those two engines, however good it is.
What this means if you own a site
None of these three show up in a rank tracker, a validator, or an uptime check. They show up when you fetch the site the way an answer engine fetches it, as the crawlers an answer engine sends, and compare that against what a browser gets.
That comparison is free here, and no scanned domain is ever named in anything we publish — including this article.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.