CrawlCheck

Findings · 2026-08-21 · By

Text extractors dropped our schema and still got us right

Two unrelated services that turn pages into text for language models both dropped the whole head of our page, and every JSON-LD node with it. Both still named the company correctly. Across ten of our own sites, 87% of our structured data sits where they do not look.

Two unrelated services that turn web pages into text for language models, a commercial scraping API and a public URL-to-markdown reader, both dropped our homepage's whole head section. Our only JSON-LD block, four nodes and 2,944 bytes, went with it, so none of it reached a model. Both still named the company correctly, because the visible text and the og: meta tags said what it is. Across ten of our own sites, 87% of schema nodes sit in the head. State every important fact in visible text as well as in schema.

We ran our homepage through two unrelated services that turn web pages into text for language models: a commercial scraping API and a public URL-to-markdown reader. Both dropped the page’s entire <head>, and with it every node of structured data we publish. Both still described the company correctly. Then we measured ten of our own sites: 87% of our structured data sits in the part those services throw away.

THE PAGE<head>JSON-LD: 4 nodes, 2,944 bytesog: and twitter: meta tags<body>visible text: what we doJSON-LD dropped: 0 nodes reach the modelmetadata object (title, description)markdown handed to the modelBoth services still named the company correctly, from the visible text and the meta tags.
What two text-extraction services kept from our homepage. The schema lived only in the head, and the head did not survive.

What did the first service return? #

The scraping API produced clean markdown at 41% of the token cost of the HTML. The HTML it returned alongside that markdown began:

<!DOCTYPE html><html lang="en"><body>

No <head>. Our homepage carries one JSON-LD block with four nodes, WebSite, SoftwareApplication, Organization and FAQPage, all in the head:

MeasureCount
JSON-LD blocks on the page1
Schema nodes in that block4
Nodes inside <head>4
Nodes inside <body>0
Nodes present in the returned HTML0

Did a second, unrelated reader do the same? #

Yes. A public URL-to-markdown reader, widely used as a drop-in for agent fetching, returned 9,742 bytes of clean markdown. We counted the same strings in its output as on the live page:

StringOn our live pageIn reader output
ld+json10
"@type"160
SoftwareApplication10
Organization10
FAQPage10
WebSite10

The whole block is 2,944 bytes and none of it survived either service. The 16 is the raw count of "@type", higher than the four top-level nodes because the FAQPage nests a Question and an Answer per entry.

Two implementations from different companies made the same call. It is not a bug in either: stripping boilerplate is what these services exist to do, and <head> is boilerplate by any reasonable definition. Two readers is enough to show this is not one vendor’s quirk. It is not a rate across the web, and we have not measured how much model-facing traffic arrives through a service like these rather than a model’s own crawler.

How did they still get us right? #

The scraping API’s structured extraction returned the company name correctly and an accurate description of the product. It got them from the visible prose and from the og: and twitter: meta tags, which it captured into a separate metadata object rather than as part of the document. So the schema was not needed here. It was simply not consulted.

Both loud versions of this story are wrong. “Structured data is dead” is wrong: search engines and answer engines read it and rich results depend on it. “These tools are broken” is wrong too: they produced accurate, clean output. The accurate version is narrower. Publishing schema and having a model see your schema are two different claims, and the second depends on what stands in between.

How much of our own schema would a head-strip remove? #

We ran the same check on the homepages of ten sites we operate: a SaaS product, an e-commerce store and eight local service businesses.

SiteSchema nodesIn head (lost)In body (kept)
SaaS product440
Tree service (A)404
E-commerce220
Fencing (A)541
Fencing (B)431
Fencing (C)330
Tree service (B)660
Concrete440
Excavation770
Landscaping660
Total4539 (87%)6 (13%)

The exception is one tree service site whose four nodes sit in <body>, only because of where a plugin happens to write them. Nobody chose that. Our own flagship, the site of the product that measures this, is in the vulnerable group.

Head is where everyone puts JSON-LD because the documentation examples do and every CMS plugin follows. To a parser that reads the whole document, position does not matter. To an extractor whose job is to discard everything that is not content, the head goes first and the schema goes with it.

Should you move your schema into the body? #

We are not recommending it:

  • It is valid but unusual, and unusual markup tends to cause a different problem later.
  • The pipelines that matter most for search still read the whole document.
  • We measured two readers on one page. That shows the behaviour exists, not how common it is.

What we do recommend is narrower. If a fact exists only in your schema, it is fragile. Every fact you would be unhappy for a model to get wrong, your name, what you do, where you serve, your phone and your hours, should be stated in visible prose as well as declared in structured data, and the two should say the same thing. That is the same discipline as a NAP consistency audit, from the other direction: there the defect is schema and page disagreeing; here it is the page not carrying the fact at all. For service-area businesses, the LocalBusiness schema guide lists which facts to put in both places.

Keep your og: and twitter: tags accurate. In this test they were the only structured signal that survived, because the extractor collected them deliberately.

Test the extraction, not the markup. A validator confirms your JSON-LD is well formed. It cannot tell you whether the reader on the other end received it. That is the same shape as a robots.txt that returns HTTP 200 with a challenge page inside: the file is fine, the validator is happy, and the machine at the far end got something else. The same reasoning applies to content that only appears after script runs, covered in the JavaScript rendering guide.

The Markdown we serve, and nobody asked for #

One more measurement that cuts against our own work. This site serves Markdown to any client that sends Accept: text/markdown, with Vary: Accept so a cache cannot mix the versions up.

Homepage representationSize
HTML (Accept: text/html)29,573 B
Markdown (Accept: text/markdown)12,030 B
Markdown as a share of HTML40%
Saved per fetch17,543 B

The scraping API did not ask for it. It sent Accept: text/html, took the HTML and converted it on its own side, arriving at 41% of the token count, within a point of the 40% our Markdown would have handed it directly. Markdown negotiation is a new convention and few clients request it yet; the Markdown for agents guide covers how to serve it and what it does and does not prove.

Both findings point the same way. Schema, meta tags, prose and negotiated representations are separate channels, and each reader takes a different subset. The only way to know which subset reached a reader is to look at what that reader actually received.

Method #

The scraping API’s output was read from its own playground, including the raw HTML it returns with the markdown. Its response was served from its cache, several hours old, so we re-checked the live page separately and relied on the cached copy only for the head-stripping behaviour, which belongs to the extractor rather than the page. Schema nodes were counted by fetching each homepage with an ordinary browser user agent, splitting at </head>, parsing every application/ld+json block and counting nodes, expanding @graph where present. Markdown and HTML sizes came from identical requests differing only in the Accept header. Homepages only; interior pages carry different graphs and were not measured.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Do text-extraction services keep JSON-LD?

Not the two we tested. A commercial scraping API and a public URL-to-markdown reader both dropped the whole head of our homepage, including our only JSON-LD block, before producing the text a model would read.

If extractors drop JSON-LD, why publish it?

Because search engines and answer engines that read the whole document use it, and rich results depend on it. Schema is the machine-checkable version of facts that should also be stated in the visible page, so that every reader gets the same answer.

How did the services still identify the company?

From the visible text and from the og: and twitter: meta tags, which one service captured into a separate metadata object. The schema was not needed; it was never consulted.

Should I move my JSON-LD into the body?

We do not recommend it. It is valid but unusual, and whole-document parsers read it in either place. The more reliable fix is to state every important fact in visible prose as well as in schema.

Where does entity identity live on a page?

In the repeated, consistent statements: the name in the title and heading, the same phone and address everywhere, the same profile links, and structured data that says the same thing. Schema is one of the sources, not the only one.

Do these services request Markdown when a site offers it?

The scraping API we tested did not. It requested HTML and converted it itself, even though our site serves Markdown to clients that ask for it.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works