CrawlCheck

Guides · 2026-10-02 · By · 1 view

Markdown for AI agents: serve text/markdown by negotiation

An agent that sends Accept: text/markdown can get a page's words without the scripts and navigation. Our measurement on four pages, the headers that must come back, and the caching trap.

Markdown for agents is HTTP content negotiation: when a request carries Accept: text/markdown, the server returns the same page as Markdown at the same URL, with Content-Type: text/markdown; charset=utf-8 and Vary: Accept. Browsers, and clients that send a wildcard Accept, keep receiving HTML. On four pages of crawlcheck.io the Markdown body was roughly a fifth to a quarter of the HTML. Vary: Accept is essential, or a cache can serve one version to the wrong client. It is a delivery improvement; no engine has said it affects ranking or citation.

Check your own site free

About a minute. First scan free with your email, nothing installed.

An agent that sends Accept: text/markdown can be given a page’s words and structure without the navigation, scripts and styles. We serve this on crawlcheck.io and measured it: on four pages, the Markdown body was roughly a fifth to a quarter of the HTML, and a client that did not specifically ask for Markdown got HTML. Below is that measurement, the headers that must come back, the caching trap, and what an edge service may add on your behalf.

GET /guides/exampleone addressAccept: text/htmlbrowser, or */*Accept: text/markdownan agent that asksHTML pagenav, scripts, stylesMarkdownthe words, structure keptVary: Accept keeps the two cached apartWithout Vary: Accept, a shared cache can hand Markdown to browsers or HTML to agents.
Same URL, two representations. The Accept header picks one; Vary: Accept stops a cache mixing them up.

What did Markdown delivery actually save? #

On 30 October 2026 we requested four paths on this site three times each: once with Accept: text/html, once with Accept: text/markdown and once with Accept: */*. All twelve responses were HTTP 200 and all advertised Vary: Accept.

PathHTML bytesMarkdown bytesSmaller by
/blog898672109476.5%
/guides/does-llms-txt-actually-work758081828075.9%
/guides/how-to-measure-ai-visibility664221536676.9%
/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites686131369080.0%
Accept sentRequestsStatusContent-Type returned
text/markdown4200text/markdown
text/html4200text/html
*/*4200text/html

The wildcard row matters more than the Markdown one. Negotiation only helps a client that sends the value the server listens for. A crawler that sends a browser-style or wildcard Accept, which is the usual crawler, lands in the HTML column with everyone else. Publishing a Markdown version and assuming agents will ask for it repeats the llms.txt mistake one layer down: the file exists, and the default client does not request it.

What this does not show: smaller bytes are not a token count, no model was in the loop, and we did not prove the Markdown keeps every fact, table and caveat of the HTML. It verifies a delivery implementation, nothing more.

How the negotiation works #

A client states the formats it prefers in the Accept request header. The server returns the matching representation of the same resource at the same URL. There is no separate address to discover and no .md suffix to guess, and nothing changes for browsers or search crawlers, which never ask for Markdown.

$ curl -sI https://crawlcheck.io/guides/xml-sitemap-audit -H 'Accept: text/markdown'
content-type: text/markdown; charset=utf-8
vary: Accept

Which response headers must come back? #

HeaderValueWhy
Content-Typetext/markdown; charset=utf-8the client can confirm it got Markdown, not HTML
Varyincludes Acceptcaches store the HTML and Markdown versions separately
Link (optional)rel="canonical" to the HTML URLkeeps one canonical address for the resource
x-markdown-tokens (Cloudflare)estimated token countlets an agent budget before it reads
content-signal (Cloudflare default)ai-train=yes, search=yes, ai-input=yes if the origin set nonea permission statement you may not have chosen

The caching trap #

A shared cache that ignores Accept lets the first client decide what is stored. If it was an agent, browsers get raw Markdown until the entry expires; if it was a browser, agents get HTML. Vary: Accept tells caches to key on the header, but some CDNs ignore Vary for cache keys unless configured. Test from outside: request the URL with and without the header, confirm each returns its own type, then repeat in reverse order after the cache has warmed. Send Vary: Accept on both representations, not only the Markdown one, or a cache that stored the HTML will never look for a second variant. Stale and split caches on machine files are a related problem, covered in the machine-file caching guide.

Converting at the edge vs at the origin #

Cloudflare’s Markdown for Agents converts HTML responses up to 2 MB at the edge on Pro, Business and Enterprise plans and sets Vary: Accept itself. It also adds a content-signal header permitting search, AI input and AI training when the origin has not sent one. That default may not match your robots.txt, so if your policy differs, send your own header from the origin; the values are explained in the content signals guide.

Doing it at the origin gives you control over what is kept. Read the Accept header; if it names text/markdown with a quality value at least as high as text/html, render Markdown, otherwise HTML. Generate the Markdown from the same stored content as the HTML rather than converting the rendered page back, so the two cannot drift. For pages with no Markdown version, such as a checkout, return the HTML with its normal content type; a 406 is also valid HTTP but breaks clients that list Markdown first with no fallback.

What belongs in the Markdown? #

Keep everything that carries meaning: title, date, author, headings in order, lists, tables, code and links with absolute URLs. Drop navigation, repeated footers, share buttons and cookie notices. Markdown has nowhere to put JSON-LD, so make sure the facts in your structured data are also in the text. And keep the words identical to the HTML: a Markdown version that says something the page does not is a different document at the same address.

Does it help visibility? #

It cuts what an agent has to fetch and removes a conversion step where tables and code are often mangled. It does not make a page rank or get cited by itself, and no engine has said it prefers sites that negotiate Markdown. In our test of two text-extraction services, neither asked for our Markdown; both took the HTML and converted it themselves. Treat it like an llms.txt file: a low-cost courtesy that helps most on pages that are mostly code.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

How does an AI agent request Markdown instead of HTML?

By sending Accept: text/markdown. A server that supports it returns the same page as Markdown at the same URL. A client that sends a browser or wildcard Accept header gets HTML.

What response headers should a Markdown response have?

Content-Type: text/markdown; charset=utf-8 and Vary: Accept, optionally with a canonical Link header pointing to the HTML URL.

Why is Vary: Accept required?

Without it, a shared cache can store whichever version was requested first and serve it to everyone, so browsers get Markdown or agents get HTML. Send it on both representations.

Do crawlers ask for Markdown on their own?

Most do not. Negotiation only helps clients that send Accept: text/markdown. Crawlers that send a browser-style or wildcard Accept header receive the HTML like everyone else.

Does Cloudflare's Markdown conversion change my AI permissions?

It adds a content-signal header allowing search, ai-input and ai-train when the origin has not set one. Send your own header from the origin if your policy differs.

Does serving Markdown improve rankings or citations?

No engine has said so. It makes the text smaller and easier for agents to use, which is a delivery improvement, not a ranking signal.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works