Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 0 views
What answer engines receive from local business sites: mostly not text
Three findings from a corpus of sites measured byte by byte, and why none of them appear in a validator.
Bars are drawn to scale. The rest of every bar is code the crawler downloads and cannot use.
Every SEO tool measures what a page says. Almost none measure what a machine is handed when it asks for that page. Those are different numbers, and the gap between them is where most of the damage lives.
How the share of text is measured
The measurement is deliberately plain. Take the bytes that arrive on the wire for one URL. Strip every script, style, noscript, template and inline svg block, drop the comments, drop the tags, collapse the whitespace. What remains is the text a client that does not run a browser could read — and, if it wanted to, quote. Divide by the bytes it cost. That ratio is stored for every page the depth crawl visits, and the site-level number reported is the median across them, so one heavy template and one light one do not cancel out into a fiction.
Two thresholds sit on that ratio. The content row of the report passes at 20%. Below 5% a separate finding fires, PAGE_IS_MOSTLY_CODE, at a weight that caps the grade — because before it existed a page at 2% and a page at 19% scored identically on the row, and they are not the same page. Across the corpus behind these figures the finding fires on roughly one scan in seven.
Live, as you read this: the corpus now holds 2,193 domains across 1,629 scans. The figures in this piece were measured on the date above; this line is not.
1. The heaviest page we measured delivered 2.3% text
476,540 bytes on the wire. Visible, readable text: 2.3% of it. The rest was markup, inline styles and scripts. A crawler pays that cost on every URL it fetches, and inline CSS cannot be cached between pages, so it pays again on the next one.
The page looked fine to every human who visited it. It renders quickly, it reads well, and no validator has an opinion about it.
2. A file returning HTTP 200 that no crawler could read
On one site, /robots.txt answered every request with a browser-verification interstitial and a 200 OK. Crawlers do not execute the JavaScript that clears such a page — they read the interstitial as the file. Every uptime monitor called the site healthy.
The cause was not the security product. It was a cache rule storing whatever came back, including a challenge page, and serving it to everyone for the next five minutes. A per-address defence became a site-wide outage of the machine layer. That one has its own write-up, including the two-request test that catches it without needing to recognise the page.
3. Two engines refused at the edge while Google walked in
On another site, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally. Nothing in robots.txt said so — this was the edge, and almost nobody configures it deliberately.
Until that is lifted, nothing else about the site reaches those two engines, however good it is. The scan records this as ANSWER_ENGINE_REFUSED, and it has two quieter siblings that only a side-by-side fetch can see: CRAWLER_SERVED_LESS, when a named crawler is handed materially less text than an ordinary client gets for the same URL, and CRAWLER_REDIRECTED_AWAY, when it is sent somewhere the browser was not. Whatever sits at that destination is what the engine reads and quotes instead of your page.
How much of a delivered page is text a machine can quote
| Page | Readable text |
|---|---|
| A page we measured | 0.8% of 2.3 MB |
| A typical local site | 18.8% |
| What we look for | 20% and up |
The rest is markup, script and style: bytes that arrive, are paid for, and say nothing an answer engine can quote.
What moved the number on four of our own sites
We run this scanner against properties we operate, and three of them sat under the 5% line. The fixes were mechanical and the results were measured, not estimated.
On the first, a page builder had emitted the same icon graphics inline dozens of times. Folding the duplicates into one symbol table and stripping comments and inter-tag whitespace took the homepage from 221,487 to 204,666 bytes with the text byte-identical, and the share from 4.78% to 5.17%. On the second, ten inline style blocks were the weight; writing each one over 2 KB out to a cached file and replacing it in place, in the same order so the cascade did not change, took 106,315 bytes to 73,133 and the share from 4.03% to 5.86%. Both crossed the line. Both are now uncapped.
The third did not, and it is the more useful case. That homepage carries 224,485 bytes and 5,485 characters of text — 2.44%. Its ten style blocks hold 73,881 bytes. Strip every one and it lands at 3.61%. Externalise all 31,000 bytes of non-schema script on top of that and it reaches about 4.5%. Still under. Compression cannot fix a page that has nothing to say; it needs words, not fewer bytes. The scanner cannot tell those two situations apart from the ratio alone, which is why the report shows the byte count and the character count side by side rather than only the percentage.
The guard that nearly shipped a silent no-op
The externaliser first refused to return any output smaller than 55% of its input, as a safety net against a transform going wrong. On the site’s 404 template that is a legitimate 48% reduction, so the guard silently did nothing there and the page kept its weight. A percentage floor is the wrong guard for a transform whose output size is computable: it now tracks what each replacement removes and adds and requires the final length to equal input minus that delta, exactly. The general lesson is the same one the viewport defect taught — a check that proves your code ran tells you nothing about whether it did the right thing.
What this means if you own a site
None of these three show up in a rank tracker, a validator, or an uptime check. They show up when you fetch the site the way an answer engine fetches it, as the crawlers an answer engine sends, and compare that against what a browser gets. The order of repair follows from the numbers: a refusal at the edge comes first, because nothing behind it is reachable; a challenge page cached as a file comes second, for the same reason; the byte ratio comes last, because a page that is reachable and thin is still reachable.
That comparison is free here, and no scanned domain is ever named in anything we publish — including this article. The terms used above are defined in the glossary.
How to see what a crawler receives from your own site
Fetch your homepage three times: as a browser, as GPTBot, and with no user-agent at all. Compare status, byte count and content type. Then strip the markup and count what is left:
for ua in 'Mozilla/5.0' 'GPTBot' ''; do curl -s -A "$ua" -o /tmp/p -w "$ua %{http_code} %{size_download}\n" https://yoursite.com/; done
sed 's/<[^>]*>/ /g' /tmp/p | wc -wThree different byte counts mean three different pages. A word count that is a small fraction of the byte count is the 2.3% problem above. How to check if AI crawlers can read your site walks through the full version; do AI crawlers render JavaScript explains why the served document is the only one that counts.
Where each of the three appears in a report
The payload section scores the text-to-bytes ratio and the render path. The machine-layer and trust-chain sections read robots.txt by content type, not status, after the 200 that could not be read. The agent-view section fetches the homepage as fifteen named identities and reports which were refused, challenged, or handed less text than an unnamed client.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
How much of a web page is text a machine can quote?
On the heaviest page measured, 2.3%. The rest was markup, script and styling that an answer engine cannot cite.
Why do validators miss this?
A validator checks whether markup is well formed. It does not weigh how much of the delivered payload is quotable text.
What should a site owner do about it?
Measure the ratio on your own heaviest pages before adding anything else. Weight you remove is quotable text you gain, proportionally.
Related findings
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.