CrawlCheck

Agent view · Guides · 2026-10-02

Markdown for AI agents: serve text/markdown by negotiation

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes16,035
words2,255
sha-256c92188fba240ee9af0aef99985a2bc77b0cbc05a5404d6aea22536818d836006
repr-digestsha-256=:ySGI+6JA7prwrvmZhaK8d7DLwFpUBNauoiU2gY2DYAY=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311481;expires=1795315081;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="j8xl1Hv01tf0EOhIh4hieA";tag="crawlcheck-page"
signaturesig1=:Pd5f025wQqvYT31jq/zxYcVMdhQvCz4g+BbBHV9yRtYeDXhjc3c6Oi6Xmb/R8kcJTWKyN43lQNSzivFl1oKwCg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-02 · By [VSNARY | Emmanuel Orta](/author) · 1 view

# Markdown for AI agents: serve text/markdown by negotiation

An agent that sends Accept: text/markdown can get a page's words without the scripts and navigation. Our measurement on four pages, the headers that must come back, and the caching trap.

Markdown for agents is HTTP content negotiation: when a request carries Accept: text/markdown, the server returns the same page as Markdown at the same URL, with Content-Type: text/markdown; charset=utf-8 and Vary: Accept. Browsers, and clients that send a wildcard Accept, keep receiving HTML. On four pages of crawlcheck.io the Markdown body was roughly a fifth to a quarter of the HTML. Vary: Accept is essential, or a cache can serve one version to the wrong client. It is a delivery improvement; no engine has said it affects ranking or citation.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What did Markdown delivery actually save?
- How the negotiation works
- Which response headers must come back?
- The caching trap
- Converting at the edge vs at the origin
- What belongs in the Markdown?
- Does it help visibility?

An agent that sends `Accept: text/markdown` can be given a page’s words and structure without the navigation, scripts and styles. We serve this on crawlcheck.io and measured it: on four pages, the Markdown body was roughly a fifth to a quarter of the HTML, and a client that did not specifically ask for Markdown got HTML. Below is that measurement, the headers that must come back, the caching trap, and what an edge service may add on your behalf.
Same URL, two representations. The Accept header picks one; Vary: Accept stops a cache mixing them up.

## What did Markdown delivery actually save? #

On 30 October 2026 we requested four paths on this site three times each: once with `Accept: text/html`, once with `Accept: text/markdown` and once with `Accept: */*`. All twelve responses were HTTP 200 and all advertised `Vary: Accept`.

| Path | HTML bytes | Markdown bytes | Smaller by |
| --- | --- | --- | --- |
| `/blog` | 89867 | 21094 | 76.5% |
| `/guides/does-llms-txt-actually-work` | 75808 | 18280 | 75.9% |
| `/guides/how-to-measure-ai-visibility` | 66422 | 15366 | 76.9% |
| `/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites` | 68613 | 13690 | 80.0% |

| Accept sent | Requests | Status | Content-Type returned |
| --- | --- | --- | --- |
| text/markdown | 4 | 200 | text/markdown |
| text/html | 4 | 200 | text/html |
| */* | 4 | 200 | text/html |

The wildcard row matters more than the Markdown one. Negotiation only helps a client that sends the value the server listens for. A crawler that sends a browser-style or wildcard `Accept`, which is the usual crawler, lands in the HTML column with everyone else. Publishing a Markdown version and assuming agents will ask for it repeats the [llms.txt](/glossary/llms-txt) mistake one layer down: the file exists, and the default client does not request it.

What this does not show: smaller bytes are not a [token](/glossary/token) count, no model was in the loop, and we did not prove the Markdown keeps every fact, table and caveat of the HTML. It verifies a delivery implementation, nothing more.

## How the negotiation works #

A client states the formats it prefers in the `Accept` request header. The server returns the matching representation of the same resource at the same [URL](/glossary/url). There is no separate address to discover and no `.md` suffix to guess, and nothing changes for browsers or search crawlers, which never ask for Markdown.

```
$ curl -sI https://crawlcheck.io/guides/xml-sitemap-audit -H 'Accept: text/markdown'
content-type: text/markdown; charset=utf-8
vary: Accept
```

## Which response headers must come back? #

| Header | Value | Why |
| --- | --- | --- |
| `Content-Type` | `text/markdown; charset=utf-8` | the client can confirm it got Markdown, not HTML |
| `Vary` | includes `Accept` | caches store the HTML and Markdown versions separately |
| `Link` (optional) | `rel="canonical"` to the HTML URL | keeps one [canonical](/glossary/canonical) address for the resource |
| `x-markdown-tokens` (Cloudflare) | estimated token count | lets an agent budget before it reads |
| `content-signal` (Cloudflare default) | `ai-train=yes, search=yes, ai-input=yes` if the origin set none | a permission statement you may not have chosen |

## The caching trap #

A shared cache that ignores `Accept` lets the first client decide what is stored. If it was an agent, browsers get raw Markdown until the entry expires; if it was a browser, agents get HTML. `Vary: Accept` tells caches to key on the header, but some CDNs ignore `Vary` for cache keys unless configured. Test from outside: request the URL with and without the header, confirm each returns its own type, then repeat in reverse order after the cache has warmed. Send `Vary: Accept` on both representations, not only the Markdown one, or a cache that stored the HTML will never look for a second variant. Stale and split caches on machine files are a related problem, covered in [the machine-file caching guide](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge).

## Converting at the edge vs at the origin #

Cloudflare’s Markdown for Agents converts HTML responses up to 2 MB at the edge on Pro, Business and Enterprise plans and sets `Vary: Accept` itself. It also adds a `content-signal` header permitting search, AI input and AI training when the origin has not sent one. That default may not match your [robots.txt](/glossary/robots-txt), so if your policy differs, send your own header from the origin; the values are explained in [the content signals guide](/guides/content-signals-in-robots-txt-search-ai-input-ai-train).

Doing it at the origin gives you control over what is kept. Read the `Accept` header; if it names `text/markdown` with a quality value at least as high as `text/html`, render Markdown, otherwise HTML. Generate the Markdown from the same stored content as the HTML rather than converting the rendered page back, so the two cannot drift. For pages with no Markdown version, such as a checkout, return the HTML with its normal content type; a 406 is also valid HTTP but breaks clients that list Markdown first with no fallback.

## What belongs in the Markdown? #

Keep everything that carries meaning: title, date, author, headings in order, lists, tables, code and links with absolute URLs. Drop navigation, repeated footers, share buttons and cookie notices. Markdown has nowhere to put [JSON-LD](/glossary/json-ld), so make sure the facts in your [structured data](/glossary/structured-data) are also in the text. And keep the words identical to the HTML: a Markdown version that says something the page does not is a different document at the same address.

## Does it help visibility? #

It cuts what an agent has to fetch and removes a conversion step where tables and code are often mangled. It does not make a page rank or get cited by itself, and no engine has said it prefers sites that negotiate Markdown. In [our test of two text-extraction services](/blog/a-pipeline-threw-away-our-schema-and-got-us-right-anyway), neither asked for our Markdown; both took the HTML and converted it themselves. Treat it like [an llms.txt file](/guides/how-to-write-an-llms-txt-and-verify-it): a low-cost courtesy that helps most on pages that are [mostly code](/guides/text-to-html-ratio-when-a-page-is-mostly-code).

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmarkdown-for-agents-serving-text-markdown-by-content-negotiation%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmarkdown-for-agents-serving-text-markdown-by-content-negotiation%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Markdown%20for%20AI%20agents%3A%20serve%20text%2Fmarkdown%20by%20negotiation)[Email](mailto:?subject=Markdown%20for%20AI%20agents%3A%20serve%20text%2Fmarkdown%20by%20negotiation&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmarkdown-for-agents-serving-text-markdown-by-content-negotiation)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How does an AI agent request Markdown instead of HTML?

By sending Accept: text/markdown. A server that supports it returns the same page as Markdown at the same URL. A client that sends a browser or wildcard Accept header gets HTML.

### What response headers should a Markdown response have?

Content-Type: text/markdown; charset=utf-8 and Vary: Accept, optionally with a canonical Link header pointing to the HTML URL.

### Why is Vary: Accept required?

Without it, a shared cache can store whichever version was requested first and serve it to everyone, so browsers get Markdown or agents get HTML. Send it on both representations.

### Do crawlers ask for Markdown on their own?

Most do not. Negotiation only helps clients that send Accept: text/markdown. Crawlers that send a browser-style or wildcard Accept header receive the HTML like everyone else.

### Does Cloudflare's Markdown conversion change my AI permissions?

It adds a content-signal header allowing search, ai-input and ai-train when the origin has not set one. Send your own header from the origin if your policy differs.

### Does serving Markdown improve rankings or citations?

No engine has said so. It makes the text smaller and easier for agents to use, which is a delivery improvement, not a ranking signal.

## Related findings

[2026-08-28 Do AI crawlers render JavaScript? Check what they see Assume AI crawlers do not run JavaScript. A one-command test shows whether your content is in the raw HTML, and why text inside a](/guides/do-ai-crawlers-render-javascript)[2026-09-30 CrawlCheck in German, French, Spanish and Portuguese CrawlCheck's main pages and every scan report now come in four more languages. What is translated, what stays English, and how the](/blog/crawlcheck-in-german-french-spanish-and-portuguese)[2026-10-02 Text-to-HTML ratio: does it matter, and how to check it Text-to-HTML ratio is no ranking factor, but a large page under 5% text costs crawlers and AI agents mostly code. Two commands to](/guides/text-to-html-ratio-when-a-page-is-mostly-code)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [ixerp.net](/registry/ixerp.net) · [partsforagents.com](/registry/partsforagents.com) · [wahidmart.com](/registry/wahidmart.com) · [Place: Daytona beach](/registry/place/fl/daytona-beach) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [TLS posture](/data/tls-posture) | **Datasets:** [Nebraska](/datasets/state/nebraska) · [Technologies: Elementor](/datasets/technologies/elementor) | **Blog:** [AI picked a local contractor 422 times: rating decided it](/blog/ai-picked-a-local-contractor-422-times-the-rating-decided-it)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/markdown-for-agents-serving-text-markdown-by-content-negotiation.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.