Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 1 view
What AI crawlers receive from local business sites
Three things we found by fetching real local business sites the way answer engines do, what moved the numbers on sites we run, and the commands to repeat it on yours.
Mostly not text. Fetched the way answer-engine crawlers fetch them, local business pages are dominated by markup, scripts and inline styling, and the visible words are a small share of the bytes. Some responses return HTTP 200 and carry nothing a crawler can use, such as a challenge page cached as robots.txt, and some sites refuse AI crawlers at the edge while serving browsers normally. Every part of the measurement is a plain fetch, so you can repeat it on your own site with curl and check the result yourself.
Bars are drawn to scale. The rest of every bar is code the crawler downloads and cannot use.
Most SEO tools measure what a page says. Almost none measure what a machine is handed when it asks for that page. We fetched local business sites as the crawlers answer engines send, byte by byte, and three things came back that no validator, rank tracker or uptime check reported.
Live, as you read this: the corpus now holds 515,593 domains across 6,674 scans. The figures in this piece were measured on the date above; this line is not.
| Finding | What the crawler got | What every monitor said |
|---|---|---|
| A heavy homepage | 476,540 bytes, of which 2.3% was readable text | Fast, renders fine |
| robots.txt behind a challenge | A browser-verification page with 200 OK, read as the file | Healthy |
| Refusal at the edge | 502 for GPTBot and ClaudeBot; Googlebot and PerplexityBot served normally | Up |
How is the share of text measured? #
Take the bytes that arrive for one URL. Strip every script, style, noscript, template and inline svg block, drop comments and tags, collapse whitespace. What remains is the text a client that does not run a browser can read and quote. Divide it by the bytes it cost. A large page that is mostly code turns up on 27.9% of scanned sites.
A homepage that was 2.3% text #
One homepage delivered 476,540 bytes, of which 2.3% was readable text. The rest was markup, inline styles and scripts. A crawler pays that cost on every URL it fetches, and inline CSS cannot be cached between pages, so it pays again on the next one. The page looked fine to every human visitor, rendered quickly, and no validator had an opinion about it. A typical local site in the same measurement delivered 18.8% text, so this was not the norm, but it was not rare either.
A robots.txt that answered 200 and could not be read #
On one site, /robots.txt answered every request with a browser-verification page and 200 OK. Crawlers do not run the JavaScript that clears such a page, so they read the interstitial as the file. The cause was not the security product but a cache rule storing whatever came back, challenge included, and serving it to everyone for the next five minutes: a per-address defence turned into a site-wide outage of the machine files. The full write-up includes the two-request test that catches it.
Two AI crawlers refused while Google walked in #
On another site, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally. Nothing in robots.txt said so; this was the edge, and almost nobody configures that deliberately. Until it is lifted, nothing else about the site reaches those two engines. It has two quieter siblings that only a side-by-side fetch shows: a crawler handed materially less text than a browser gets for the same URL, and a crawler redirected somewhere the browser was not. The delivery comparison guide covers all three.
What do the bytes break down into? #
Text share is a summary. The useful question is which category ate the rest:
| Category | What it is | Can an engine quote it? |
|---|---|---|
| Readable text | The words a reader sees | Yes; this is the entire quotable surface |
| Markup and attributes | Tags, classes, data attributes | No |
| Structured data | JSON-LD blocks | Not as prose, but it is how facts attach to an entity |
| Inline CSS | Styles in the document rather than a file | No, and it cannot be cached between pages |
| Inline JavaScript | Scripts in the document | No |
| Inline SVG | Vector graphics written into the HTML | No, and it is often the largest single block |
| HTML comments | Notes, build stamps, commented-out sections | No, and they ship to every client |
Inline SVG is the one that surprises people. An icon set written into the document instead of referenced as a file can outweigh the article beside it, and it compresses well enough that a page-weight audit does not notice.
What moved the number on sites we run #
We scan properties we operate, and three of them carried very little text. On the first, a page builder had emitted the same icon graphics inline dozens of times. Folding the duplicates into one symbol table and stripping comments and inter-tag whitespace took the homepage from 221,487 to 204,666 bytes with the text unchanged, and the share from 4.78% to 5.17%. On the second, ten inline style blocks were the weight; writing each one over 2 KB out to a cached file, in the same order so the cascade did not change, took 106,315 bytes to 73,133 and the share from 4.03% to 5.86%.
The third did not, and it is the more useful case. That homepage carries 224,485 bytes and 5,485 characters of text, 2.44%. Its ten style blocks hold 73,881 bytes; strip every one and it lands at 3.61%. Externalise all 31,000 bytes of non-schema script on top and it reaches about 4.5%. Still thin. A page that has too little to say needs words, not fewer bytes. A ratio alone cannot tell those two situations apart, so always read the byte count and the character count together.
How do I check my own site? #
Set ua to your own browser's user-agent string, then run:
# The same URL as a browser and as an answer-engine crawler
curl -s --compressed -o browser.html -w '%{http_code} %{size_download}\n' -A "$ua" https://example.com/
curl -s --compressed -o crawler.html -w '%{http_code} %{size_download}\n' -A "GPTBot" https://example.com/
# The control: the identical browser request again
curl -s --compressed -o browser2.html -w '%{http_code} %{size_download}\n' -A "$ua" https://example.com/
# Share of the delivered bytes that is readable text
wc -c < crawler.html
perl -0pe 's/<(script|style|noscript|template|svg)\b.*?<\/\1>//gis; s/<!--.*?-->//gs; s/<[^>]+>//g; s/\s+/ /g' crawler.html | wc -cIf the crawler's size differs from the browser's by more than the two browser requests differ from each other, something in your stack is deciding what to serve based on who asked: an inherited CDN rule, a bot-management product, or a cache that splits on user agent. The control matters. A site that serves 90 KB to one browser request and 45 KB to an identical one a second later, because of a split test, a rotating banner or a cold cache, has not discriminated against a crawler that got 40 KB; the gap is inside its own noise. Repeat with ClaudeBot and PerplexityBot, and from a second network before you call a refusal a block on crawlers. The text-to-HTML ratio guide shows how to find the largest block to remove.
What this measurement cannot see #
Everything above is what arrives before any JavaScript runs, which is what a crawler fetching your URL receives. A page can be 3% text on the wire and perfectly readable in a browser, and a text-rich page can still fail for a crawler refused at the edge. These are independent failures; treating a low text share as proof of invisibility, or a high one as proof of health, is the usual mistake. Our scan reads what a non-browser client is served from a datacenter address, and the browser extension reads what is served to your own address after your scripts run. Where the two disagree is where the answer usually is.
What to fix first #
A refusal at the edge comes first, because nothing behind it is reachable. A challenge page cached as a machine file comes second, for the same reason. The byte ratio comes last: a page that is reachable and thin is still reachable. This article names no site; the terms used are defined in the glossary.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How much of a web page is text a machine can quote?
Often far less than people assume. On one local business homepage we measured, 2.3% of the delivered bytes were readable text; the rest was markup, script and styling that an answer engine cannot cite.
Why do validators miss this?
A validator checks whether markup is well formed. It does not weigh how much of the delivered payload is quotable text, or whether a crawler was served the page at all.
How do I measure the text share of my own page?
Fetch the page with curl, count the bytes, strip scripts, styles, SVG, comments and tags, and count what is left. The second number over the first is the share of what you delivered that a machine can quote.
Do AI crawlers get the same page a browser gets?
Usually, but not always, and the exceptions are invisible from a browser. Request the same URL as a browser twice and as a named crawler once, from the same address, and compare sizes. A crawler difference only means something when it is larger than the difference between the two browser requests.
Why does a page return 200 and still give a crawler nothing?
A status code describes the transaction, not the content. A 200 can carry a challenge page, an empty shell awaiting JavaScript, a soft error page or a machine file served as HTML, and all of them look like success on a monitoring dashboard.
Does removing code always fix a low text share?
No. If a page has very few words, stripping styles and scripts raises the share only a little. Such a page needs more content, not fewer bytes.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.