Agent view · Findings · 2026-08-13

What AI crawlers receive from local business sites

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/what-a-crawler-actually-receives: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-08-13 · By [VSNARY | Emmanuel Orta](/author) · 1 view

# What AI crawlers receive from local business sites

Three things we found by fetching real local business sites the way answer engines do, what moved the numbers on sites we run, and the commands to repeat it on yours.

Mostly not text. Fetched the way answer-engine crawlers fetch them, local business pages are dominated by markup, scripts and inline styling, and the visible words are a small share of the bytes. Some responses return HTTP 200 and carry nothing a crawler can use, such as a challenge page cached as robots.txt, and some sites refuse AI crawlers at the edge while serving browsers normally. Every part of the measurement is a plain fetch, so you can repeat it on your own site with curl and check the result yourself.

On this page

- How is the share of text measured?
- A homepage that was 2.3% text
- A robots.txt that answered 200 and could not be read
- Two AI crawlers refused while Google walked in
- What do the bytes break down into?
- What moved the number on sites we run
- How do I check my own site?
- What this measurement cannot see
- What to fix first

How much of a delivered page is text a machine can quote
A page we measured 0.8% of 2.3MB

A typical local site 18.8%

What we look for 20% and up

Bars are drawn to scale. The rest of every bar is code the crawler downloads and cannot use.

Most SEO tools measure what a page *says*. Almost none measure what a machine is *handed* when it asks for that page. We fetched local business sites as the crawlers answer engines send, byte by byte, and three things came back that no validator, rank tracker or uptime check reported.

**Live, as you read this:** the corpus now holds 515,593 domains across 6,674 scans. The figures in this piece were measured on the date above; this line is not.

| Finding | What the crawler got | What every monitor said |
| --- | --- | --- |
| A heavy homepage | 476,540 bytes, of which 2.3% was readable text | Fast, renders fine |
| [robots.txt](/glossary/robots-txt) behind a challenge | A browser-verification page with 200 OK, read as the file | Healthy |
| [Refusal](/glossary/refusal) at the edge | 502 for GPTBot and ClaudeBot; Googlebot and PerplexityBot served normally | Up |

## How is the share of text measured? #

Take the bytes that arrive for one [URL](/glossary/url). Strip every `script`, `style`, `noscript`, `template` and inline `svg` block, drop comments and tags, collapse whitespace. What remains is the text a client that does not run a browser can read and quote. Divide it by the bytes it cost. A large page that is mostly code turns up on 27.9% of scanned sites.

## A homepage that was 2.3% text #

One homepage delivered 476,540 bytes, of which **2.3%** was readable text. The rest was markup, inline styles and scripts. A crawler pays that cost on every URL it fetches, and inline CSS cannot be cached between pages, so it pays again on the next one. The page looked fine to every human visitor, rendered quickly, and no validator had an opinion about it. A typical local site in the same measurement delivered 18.8% text, so this was not the norm, but it was not rare either.

## A robots.txt that answered 200 and could not be read #

On one site, `/robots.txt` answered every request with a browser-verification page and **200 OK**. Crawlers do not run the JavaScript that clears such a page, so they read the interstitial as the file. The cause was not the security product but a cache rule storing whatever came back, challenge included, and serving it to everyone for the next five minutes: a per-address defence turned into a site-wide outage of the machine files. [The full write-up](/blog/a-file-that-answered-200-and-could-not-be-read) includes the two-request test that catches it.

## Two AI crawlers refused while Google walked in #

On another site, GPTBot and ClaudeBot both received **502** while Googlebot and PerplexityBot were served normally. Nothing in robots.txt said so; this was the edge, and almost nobody configures that deliberately. Until it is lifted, nothing else about the site reaches those two engines. It has two quieter siblings that only a side-by-side fetch shows: a crawler handed materially less text than a browser gets for the same URL, and a crawler redirected somewhere the browser was not. [The delivery comparison guide](/guides/crawler-delivery-comparison-refused-redirected-or-served-less) covers all three.

## What do the bytes break down into? #

Text share is a summary. The useful question is which category ate the rest:

| Category | What it is | Can an engine quote it? |
| --- | --- | --- |
| Readable text | The words a reader sees | Yes; this is the entire quotable surface |
| Markup and attributes | Tags, classes, data attributes | No |
| [Structured data](/glossary/structured-data) | [JSON-LD](/glossary/json-ld) blocks | Not as prose, but it is how facts attach to an entity |
| Inline CSS | Styles in the document rather than a file | No, and it cannot be cached between pages |
| Inline JavaScript | Scripts in the document | No |
| Inline SVG | Vector graphics written into the HTML | No, and it is often the largest single block |
| HTML comments | Notes, build stamps, commented-out sections | No, and they ship to every client |

Inline SVG is the one that surprises people. An icon set written into the document instead of referenced as a file can outweigh the article beside it, and it compresses well enough that a page-weight audit does not notice.

## What moved the number on sites we run #

We scan properties we operate, and three of them carried very little text. On the first, a page builder had emitted the same icon graphics inline dozens of times. Folding the duplicates into one symbol table and stripping comments and inter-tag whitespace took the homepage from 221,487 to 204,666 bytes with the text unchanged, and the share from **4.78% to 5.17%**. On the second, ten inline style blocks were the weight; writing each one over 2 KB out to a cached file, in the same order so the cascade did not change, took 106,315 bytes to 73,133 and the share from **4.03% to 5.86%**.

The third did not, and it is the more useful case. That homepage carries 224,485 bytes and **5,485 characters of text**, 2.44%. Its ten style blocks hold 73,881 bytes; strip every one and it lands at 3.61%. Externalise all 31,000 bytes of non-schema script on top and it reaches about 4.5%. Still thin. **A page that has too little to say needs words, not fewer bytes.** A ratio alone cannot tell those two situations apart, so always read the byte count and the character count together.
Removing bytes helped the first two. The third had too few words; compression cannot fix that.

## How do I check my own site? #

Set `ua` to your own browser's user-agent string, then run:

```
# The same URL as a browser and as an answer-engine crawler
curl -s --compressed -o browser.html -w '%{http_code} %{size_download}\n' -A "$ua" https://example.com/
curl -s --compressed -o crawler.html -w '%{http_code} %{size_download}\n' -A "GPTBot" https://example.com/

# The control: the identical browser request again
curl -s --compressed -o browser2.html -w '%{http_code} %{size_download}\n' -A "$ua" https://example.com/

# Share of the delivered bytes that is readable text
wc -c < crawler.html
perl -0pe 's/<(script|style|noscript|template|svg)\b.*?<\/\1>//gis; s/<!--.*?-->//gs; s/<[^>]+>//g; s/\s+/ /g' crawler.html | wc -c
```

If the crawler's size differs from the browser's by more than the two browser requests differ from each other, something in your stack is deciding what to serve based on who asked: an inherited [CDN](/glossary/cdn) rule, a bot-management product, or a cache that splits on [user agent](/glossary/user-agent). The control matters. A site that serves 90 KB to one browser request and 45 KB to an identical one a second later, because of a split test, a rotating banner or a cold cache, has not discriminated against a crawler that got 40 KB; the gap is inside its own noise. Repeat with `ClaudeBot` and `PerplexityBot`, and from a second network before you call a refusal a block on crawlers. [The text-to-HTML ratio guide](/guides/text-to-html-ratio-when-a-page-is-mostly-code) shows how to find the largest block to remove.

## What this measurement cannot see #

Everything above is what arrives before any JavaScript runs, which is what a crawler fetching your URL receives. A page can be 3% text on the wire and perfectly readable in a browser, and a text-rich page can still fail for a crawler refused at the edge. These are independent failures; treating a low text share as proof of invisibility, or a high one as proof of health, is the usual mistake. Our scan reads what a non-browser client is served from a datacenter address, and the browser extension reads what is served to your own address after your scripts run. Where the two disagree is where the answer usually is.

## What to fix first #

A refusal at the edge comes first, because nothing behind it is reachable. A challenge page cached as a machine file comes second, for the same reason. The byte ratio comes last: a page that is reachable and thin is still reachable. This article names no site; the terms used are defined in the [glossary](/glossary).

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwhat-a-crawler-actually-receives%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwhat-a-crawler-actually-receives%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=What%20AI%20crawlers%20receive%20from%20local%20business%20sites)[Email](mailto:?subject=What%20AI%20crawlers%20receive%20from%20local%20business%20sites&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwhat-a-crawler-actually-receives)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How much of a web page is text a machine can quote?

Often far less than people assume. On one local business homepage we measured, 2.3% of the delivered bytes were readable text; the rest was markup, script and styling that an answer engine cannot cite.

### Why do validators miss this?

A validator checks whether markup is well formed. It does not weigh how much of the delivered payload is quotable text, or whether a crawler was served the page at all.

### How do I measure the text share of my own page?

Fetch the page with curl, count the bytes, strip scripts, styles, SVG, comments and tags, and count what is left. The second number over the first is the share of what you delivered that a machine can quote.

### Do AI crawlers get the same page a browser gets?

Usually, but not always, and the exceptions are invisible from a browser. Request the same URL as a browser twice and as a named crawler once, from the same address, and compare sizes. A crawler difference only means something when it is larger than the difference between the two browser requests.

### Why does a page return 200 and still give a crawler nothing?

A status code describes the transaction, not the content. A 200 can carry a challenge page, an empty shell awaiting JavaScript, a soft error page or a machine file served as HTML, and all of them look like success on a monitoring dashboard.

### Does removing code always fix a low text share?

No. If a page has very few words, stripping styles and scripts raises the share only a little. Such a page needs more content, not fewer bytes.

## Related findings

[2026-09-16 HTTP status codes for crawlers: what a 200 can hide A 200 can carry a bot challenge and a 403 can be a wall on every path. How to read status codes the way crawlers get them, with cu](/guides/http-status-codes-what-a-crawler-actually-gets)[2026-09-05 areaServed vs service area pages: 20 towns declared, 1 page Four local sites we run declare 19 to 23 service areas in schema. Three have pages for 7, 1 and 1. Why that gap matters and how to](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them)[2026-10-03 Rich Results Test vs Schema Markup Validator vs CrawlCheck Rich Results Test, Schema Markup Validator and CrawlCheck compared: what each checks, valid markup that contradicts the page, and](/guides/schema-validators-compared-rich-results-test-vs-schema-markup-validator)

Where this is used

Guides that cite this finding:

- [Do AI crawlers render JavaScript? Check what they see](/guides/do-ai-crawlers-render-javascript)
- [GEO vs AEO vs LLM SEO: the terms, and what is measurable](/guides/geo-aeo-llm-seo-what-the-terms-mean)
- [How to check if AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [SDK](/sdk) · [Certificates](/certificates) · [Conduct: Google notebooklm](/agents/conduct/google-notebooklm) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [LLMs](/registry/llms) | **AI crawlers:** [Applebot extended vs Google-Extended](/ai-crawlers/compare/applebot-extended-vs-google-extended) | **Guides:** [Indexnow what it does who reads it and how to check it worked](/guides/indexnow-what-it-does-who-reads-it-and-how-to-check-it-worked) | **Glossary:** [training data](/glossary/training-data) · [Cosine similarity](/glossary/cosine-similarity) · [GET request](/glossary/get-request) · [Sponsored link](/glossary/sponsored-link) · [Jailbreak](/glossary/jailbreak)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/what-a-crawler-actually-receives.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.