Agent view · Findings · 2026-08-21
Text extractors dropped our schema and still got us right
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 18,138 |
| words | 2,729 |
| sha-256 | 42e2f416682dfdf960e97c273c2fbaeb7f0cfd3f95a3ba2ee9162d3fd32222b3 |
| repr-digest | sha-256=:QuL0Fmgt/flg6XwnPC+6638M/T+Vo7ou6RYtP9MiIrM=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311496;expires=1795315096;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="8l2dNGUm8SPpFQYg3m7Uqw";tag="crawlcheck-page" |
| signature | sig1=:32nXcr44kcsDZjmcj9zcvOXUBcoFDNQt4k2lwz0A9NkRmPgtekDxv/7S55YMv9zyxcy+RNfrSNPJVmvpsXJrBg==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-08-21 · By [VSNARY | Emmanuel Orta](/author) # Text extractors dropped our schema and still got us right Two unrelated services that turn pages into text for language models both dropped the whole head of our page, and every JSON-LD node with it. Both still named the company correctly. Across ten of our own sites, 87% of our structured data sits where they do not look. Two unrelated services that turn web pages into text for language models, a commercial scraping API and a public URL-to-markdown reader, both dropped our homepage's whole head section. Our only JSON-LD block, four nodes and 2,944 bytes, went with it, so none of it reached a model. Both still named the company correctly, because the visible text and the og: meta tags said what it is. Across ten of our own sites, 87% of schema nodes sit in the head. State every important fact in visible text as well as in schema. On this page - What did the first service return? - Did a second, unrelated reader do the same? - How did they still get us right? - How much of our own schema would a head-strip remove? - Should you move your schema into the body? - The Markdown we serve, and nobody asked for - Method We ran our homepage through two unrelated services that turn web pages into text for language models: a commercial scraping API and a public URL-to-markdown reader. Both dropped the page’s entire `<head>`, and with it every node of [structured data](/glossary/structured-data) we publish. Both still described the company correctly. Then we measured ten of our own sites: 87% of our structured data sits in the part those services throw away. What two text-extraction services kept from our homepage. The schema lived only in the head, and the head did not survive. ## What did the first service return? # The scraping API produced clean markdown at 41% of the [token](/glossary/token) cost of the HTML. The HTML it returned alongside that markdown began: ``` <!DOCTYPE html><html lang="en"><body> ``` No `<head>`. Our homepage carries one [JSON-LD](/glossary/json-ld) block with four nodes, `WebSite`, `SoftwareApplication`, `Organization` and `FAQPage`, all in the head: | Measure | Count | | --- | --- | | JSON-LD blocks on the page | 1 | | Schema nodes in that block | 4 | | Nodes inside `<head>` | 4 | | Nodes inside `<body>` | 0 | | **Nodes present in the returned HTML** | **0** | ## Did a second, unrelated reader do the same? # Yes. A public URL-to-markdown reader, widely used as a drop-in for agent fetching, returned 9,742 bytes of clean markdown. We counted the same strings in its output as on the live page: | String | On our live page | In reader output | | --- | --- | --- | | `ld+json` | 1 | **0** | | `"@type"` | 16 | **0** | | `SoftwareApplication` | 1 | **0** | | `Organization` | 1 | **0** | | `FAQPage` | 1 | **0** | | `WebSite` | 1 | **0** | The whole block is 2,944 bytes and none of it survived either service. The 16 is the raw count of `"@type"`, higher than the four top-level nodes because the `FAQPage` nests a `Question` and an `Answer` per entry. Two implementations from different companies made the same call. It is not a bug in either: stripping [boilerplate](/glossary/boilerplate) is what these services exist to do, and `<head>` is boilerplate by any reasonable definition. Two readers is enough to show this is not one vendor’s quirk. It is not a rate across the web, and we have not measured how much model-facing traffic arrives through a service like these rather than a model’s own crawler. ## How did they still get us right? # The scraping API’s structured extraction returned the company name correctly and an accurate description of the product. It got them from the visible prose and from the `og:` and `twitter:` meta tags, which it captured into a separate metadata object rather than as part of the document. So the schema was not needed here. It was simply not consulted. Both loud versions of this story are wrong. “Structured data is dead” is wrong: search engines and answer engines read it and rich results depend on it. “These tools are broken” is wrong too: they produced accurate, clean output. The accurate version is narrower. **Publishing schema and having a model see your schema are two different claims, and the second depends on what stands in between.** ## How much of our own schema would a head-strip remove? # We ran the same check on the homepages of ten sites we operate: a SaaS product, an e-commerce store and eight local service businesses. | Site | Schema nodes | In head (lost) | In body (kept) | | --- | --- | --- | --- | | SaaS product | 4 | 4 | 0 | | Tree service (A) | 4 | 0 | 4 | | E-commerce | 2 | 2 | 0 | | Fencing (A) | 5 | 4 | 1 | | Fencing (B) | 4 | 3 | 1 | | Fencing (C) | 3 | 3 | 0 | | Tree service (B) | 6 | 6 | 0 | | Concrete | 4 | 4 | 0 | | Excavation | 7 | 7 | 0 | | Landscaping | 6 | 6 | 0 | | **Total** | **45** | **39 (87%)** | **6 (13%)** | The exception is one tree service site whose four nodes sit in `<body>`, only because of where a plugin happens to write them. Nobody chose that. Our own flagship, the site of the product that measures this, is in the vulnerable group. Head is where everyone puts JSON-LD because the documentation examples do and every CMS plugin follows. To a parser that reads the whole document, position does not matter. To an extractor whose job is to discard everything that is not content, the head goes first and the schema goes with it. ## Should you move your schema into the body? # We are not recommending it: - It is valid but unusual, and unusual markup tends to cause a different problem later. - The pipelines that matter most for search still read the whole document. - We measured two readers on one page. That shows the behaviour exists, not how common it is. What we do recommend is narrower. **If a fact exists only in your schema, it is fragile.** Every fact you would be unhappy for a model to get wrong, your name, what you do, where you serve, your phone and your hours, should be stated in visible prose as well as declared in structured data, and the two should say the same thing. That is the same discipline as a [NAP consistency audit](/guides/nap-consistency-audit-phone-address-and-coordinates), from the other direction: there the defect is schema and page disagreeing; here it is the page not carrying the fact at all. For service-area businesses, [the LocalBusiness schema guide](/guides/localbusiness-schema-for-service-area-businesses) lists which facts to put in both places. **Keep your og: and twitter: tags accurate.** In this test they were the only structured signal that survived, because the extractor collected them deliberately. **Test the extraction, not the markup.** A validator confirms your JSON-LD is well formed. It cannot tell you whether the reader on the other end received it. That is the same shape as a [robots.txt](/glossary/robots-txt) that returns HTTP 200 with a challenge page inside: the file is fine, the validator is happy, and the machine at the far end got something else. The same reasoning applies to content that only appears after script runs, covered in [the JavaScript rendering guide](/guides/do-ai-crawlers-render-javascript). ## The Markdown we serve, and nobody asked for # One more measurement that cuts against our own work. This site serves Markdown to any client that sends `Accept: text/markdown`, with `Vary: Accept` so a cache cannot mix the versions up. | Homepage representation | Size | | --- | --- | | HTML (`Accept: text/html`) | 29,573 B | | Markdown (`Accept: text/markdown`) | 12,030 B | | Markdown as a share of HTML | **40%** | | Saved per fetch | 17,543 B | The scraping API did not ask for it. It sent `Accept: text/html`, took the HTML and converted it on its own side, arriving at 41% of the token count, within a point of the 40% our Markdown would have handed it directly. [Markdown negotiation](/glossary/markdown-negotiation) is a new convention and few clients request it yet; the [Markdown for agents guide](/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation) covers how to serve it and what it does and does not prove. Both findings point the same way. Schema, meta tags, prose and negotiated representations are separate channels, and each reader takes a different subset. The only way to know which subset reached a reader is to look at what that reader actually received. ## Method # The scraping API’s output was read from its own playground, including the raw HTML it returns with the markdown. Its response was served from its cache, several hours old, so we re-checked the live page separately and relied on the cached copy only for the head-stripping behaviour, which belongs to the extractor rather than the page. Schema nodes were counted by fetching each homepage with an ordinary browser [user agent](/glossary/user-agent), splitting at `</head>`, parsing every `application/ld+json` block and counting nodes, expanding `@graph` where present. Markdown and HTML sizes came from identical requests differing only in the `Accept` header. Homepages only; interior pages carry different graphs and were not measured. CrawlCheck’s dataset when this page loaded: 582,334 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-pipeline-threw-away-our-schema-and-got-us-right-anyway%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-pipeline-threw-away-our-schema-and-got-us-right-anyway%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Text%20extractors%20dropped%20our%20schema%20and%20still%20got%20us%20right)[Email](mailto:?subject=Text%20extractors%20dropped%20our%20schema%20and%20still%20got%20us%20right&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-pipeline-threw-away-our-schema-and-got-us-right-anyway)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Do text-extraction services keep JSON-LD? Not the two we tested. A commercial scraping API and a public URL-to-markdown reader both dropped the whole head of our homepage, including our only JSON-LD block, before producing the text a model would read. ### If extractors drop JSON-LD, why publish it? Because search engines and answer engines that read the whole document use it, and rich results depend on it. Schema is the machine-checkable version of facts that should also be stated in the visible page, so that every reader gets the same answer. ### How did the services still identify the company? From the visible text and from the og: and twitter: meta tags, which one service captured into a separate metadata object. The schema was not needed; it was never consulted. ### Should I move my JSON-LD into the body? We do not recommend it. It is valid but unusual, and whole-document parsers read it in either place. The more reliable fix is to state every important fact in visible prose as well as in schema. ### Where does entity identity live on a page? In the repeated, consistent statements: the name in the title and heading, the same phone and address everywhere, the same profile links, and structured data that says the same thing. Schema is one of the sources, not the only one. ### Do these services request Markdown when a site offers it? The scraping API we tested did not. It requested HTML and converted it itself, even though our site serves Markdown to clients that ask for it. ## Related findings [2026-09-05 areaServed vs service area pages: 20 towns declared, 1 page Four local sites we run declare 19 to 23 service areas in schema. Three have pages for 7, 1 and 1. Why that gap matters and how to](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them)[2026-09-16 Brand schema for multi-location businesses: link both ways A parent site listed twelve businesses under its brand; the locations never pointed back. How to declare brand from each location,](/blog/the-parent-declared-the-chain-none-of-the-locations-declared-it-back)[2026-08-14 MobileOptimized tag: on 1.65M sites, none in the top 10,000 MobileOptimized is on about 1.65 million sites and rising, but absent from the top 10,000. How to read an adoption count by rank b](/blog/the-meta-tag-a-million-sites-have-and-the-top-ten-thousand-do-not) Where this is used Guides that cite this finding: - [Do AI crawlers render JavaScript? Check what they see](/guides/do-ai-crawlers-render-javascript) - [What is entity SEO? What machines must resolve to cite you](/guides/what-is-entity-seo) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) | **Glossary:** [stapled verdict](/glossary/stapled-verdict) · [Referrer-Policy](/glossary/referrer-policy) | **Registry:** [app.alkhemyst-ai.cloud](/registry/app.alkhemyst-ai.cloud) · [dailymotion.com](/registry/dailymotion.com) · [haskell.org](/registry/haskell.org) · [mikeerdmancadillac.com](/registry/mikeerdmancadillac.com) · [terrariumstation.com](/registry/terrariumstation.com) · [Place: Orlando](/registry/place/fl/orlando) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.