CrawlCheck

Agent view · Guides · 2026-10-03

Best llms.txt generators and validators compared

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/best-llms-txt-generators-and-validators-compared: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/best-llms-txt-generators-and-validators-compared>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes15,237
words2,108
sha-25654483a19d6e1c5d2a76250374a27a1f1ac675ad8dd60feae0c15f7b0de347dc8
repr-digestsha-256=:VEg6GdbhxdKnYlA3Sieh8axnWtjdYP6uDBX3sN40fcg=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311486;expires=1795315086;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="8o7UFTGQ4imdFeiS2R7rYg";tag="crawlcheck-page"
signaturesig1=:fVI4mtiI4eafqTedpTrzo3Wm0Cj+hhBtS2Mfg/YTUF9rCO0Pjvt8eqM7litra8n0ADl/WTwQnknbAFFYAl8uAw==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 3 views

# Best llms.txt generators and validators compared

Yoast, Mintlify, Firecrawl and CrawlCheck compared on writing llms.txt, keeping it current, and checking that AI crawlers actually receive it.

Yoast SEO generates and refreshes llms.txt automatically for WordPress and Shopify, free. Mintlify hosts llms.txt and llms-full.txt for documentation sites. Firecrawl's free generator builds llms.txt and llms-full.txt for any site, and CrawlCheck drafts one from your declared pages. None of the generators checks the published file. CrawlCheck's free scan does: it fetches the file as AI crawlers do and reports whether they get text, a current copy and the same file on every host.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- Which tools generate llms.txt, and which check it?
- What does a generator not check?
- How common are missing and stale files?
- Which tool should you use?
- Where is CrawlCheck weaker?
- Is llms.txt worth publishing at all?

An [llms.txt](/glossary/llms-txt) is a Markdown file at the root of a site that points AI tools to its most useful pages. The [proposal at llmstxt.org](https://llmstxt.org/) (by Jeremy Howard) requires only an H1 with the site’s name. Writing one is now easy: several tools do it automatically. What none of the generators checks is whether the published file actually reaches AI crawlers as text. The format itself is covered in [how to write an llms.txt and verify it](/guides/how-to-write-an-llms-txt-and-verify-it); this page compares the tools.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every claim about another company comes from that company’s own pricing or documentation page, linked below, read on 3 October 2026. Where CrawlCheck does less than a competitor, the table says so.

## Which tools generate llms.txt, and which check it? #

| Tool | What it does | Keeps it updated? | Checks it is served to crawlers? | Price |
| --- | --- | --- | --- | --- |
| **CrawlCheck** | Drafts an llms.txt from your homepage and declared pages; audits the live file | No (draft only) | Yes: fetches /llms.txt on apex and www as AI crawlers do and reports status, content type, freshness and whether the pages it lists resolve | Free |
| Firecrawl llms.txt generator | Crawls a site and generates llms.txt and [llms-full.txt](/glossary/llms-full-txt) (full page content) | No; regenerate on demand | No | Free; a free [API key](/glossary/api-key) removes limits |
| Yoast SEO | Generates llms.txt for WordPress and Shopify sites, automatic or manual mode | Yes, automatically | No | Included free in Yoast |
| Mintlify | Hosts llms.txt and llms-full.txt for documentation sites | Yes, automatically | No | Part of Mintlify docs hosting |

Sources: [Firecrawl](https://www.firecrawl.dev/blog/How-to-Create-an-llms-txt-File-for-Any-Website), [Yoast](https://yoast.com/features/llms-txt/), [Mintlify](https://www.mintlify.com/docs/ai/llmstxt), and the free CrawlCheck [scan](/).

## What does a generator not check? #

Generating the file and delivering it are separate steps; no generator in this comparison checks the second.
A file that exists in your CMS can still fail to reach a crawler, and every one of these failures looks fine in a browser. The same pattern on [robots.txt](/glossary/robots-txt), a challenge page answered with 200, is in [a file that answered 200 and could not be read](/blog/a-file-that-answered-200-and-could-not-be-read); the cache cases are in [the machine-file caching guide](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge).

| Failure | What a crawler gets | How to see it |
| --- | --- | --- |
| Page template at /llms.txt | HTML or a login wall, with status 200 | `curl -sI`: content type is `text/html` |
| Stale cache | The copy from before your last edit | Request with and without a cache-busting query; compare |
| Apex and www differ | Whichever file the host it started on serves | Fetch both hosts; compare |
| Bot rule | A [refusal](/glossary/refusal) the browser never sees | Fetch as each crawler, not as a browser |

## How common are missing and stale files? #

Two live figures from CrawlCheck scans. Both count sites people chose to scan with CrawlCheck, not a fixed list of top domains, so they answer a different question from a census of the web and the two shares should not be compared directly. Share of scanned sites with no llms.txt at all: **34.1%**.

Share of scanned sites where a cache serves crawlers an out-of-date machine file: **0.4%**.

## Which tool should you use? #

- **WordPress or Shopify on Yoast:** turn on Yoast’s llms.txt; it updates itself.
- **Documentation on Mintlify:** it is already there.
- **Any other site:** generate a draft with Firecrawl’s generator or CrawlCheck’s free drafter, cut it to the pages that actually answer customers’ questions, and publish it.
- **Every site, after publishing:** run a [free scan](/) to confirm AI crawlers receive the file as text.

The first check takes one command: `curl -sI https://example.com/llms.txt`, then the same for the www host, reading the status line and the `content-type` header. The full post-publish checklist is in the [writing guide](/guides/how-to-write-an-llms-txt-and-verify-it).

## Where is CrawlCheck weaker? #

CrawlCheck’s drafter works from your homepage and declared pages and does not keep the file updated; Yoast and Mintlify do that automatically. Firecrawl also produces llms-full.txt with full page content, which CrawlCheck does not. CrawlCheck’s strength is the check after publishing.

## Is llms.txt worth publishing at all? #

It is a proposal, not a standard the major AI crawlers have committed to read. On our own site, where every page links it, sixteen crawler names fetched it 8 times in two weeks, about as often as an ordinary blog post ([the measurement](/blog/our-llms-txt-is-linked-from-every-page-crawlers-fetch-it-like-a-blog-post)). Treat it as low-cost and low-certainty: worth publishing correctly, not worth publishing broken, and never the only place an important fact appears. The wider evidence is in [does llms.txt actually work?](/guides/does-llms-txt-actually-work)
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-llms-txt-generators-and-validators-compared%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-llms-txt-generators-and-validators-compared%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Best%20llms.txt%20generators%20and%20validators%20compared)[Email](mailto:?subject=Best%20llms.txt%20generators%20and%20validators%20compared&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fbest-llms-txt-generators-and-validators-compared)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is the best llms.txt generator?

For WordPress or Shopify sites on Yoast SEO, Yoast's built-in llms.txt is free and updates automatically. Mintlify documentation sites get llms.txt and llms-full.txt automatically. For any other site, Firecrawl's free generator and CrawlCheck's free drafter create a starting file. Whatever you use, confirm the live file is served to AI crawlers as text.

### Is there a free llms.txt validator?

Yes. CrawlCheck's free llms.txt audit checks /llms.txt on the apex and www hosts as AI crawlers receive it: status, content type, cache freshness and whether the pages it cites resolve. A file that answers 200 with HTML or a login wall is not a working llms.txt.

### What is the difference between llms.txt and llms-full.txt?

llms.txt is a short Markdown index with links to a site's key pages. llms-full.txt, produced by tools such as Firecrawl and Mintlify, includes the full content of each page in one file.

### Does Yoast create llms.txt?

Yes. Yoast's llms.txt feature is free in Yoast SEO, Premium, WooCommerce SEO, AI+ and Yoast SEO for Shopify, and generates and refreshes the file automatically, with a manual mode for choosing content.

### Why does my llms.txt show a web page instead of text?

Usually because the CMS routes unknown paths to a page template, or a login or bot challenge sits in front of the file, and both can answer with status 200. Check the content-type header: a working llms.txt is served as text/plain or text/markdown, not text/html.

### Do AI crawlers read llms.txt?

It is a proposal rather than a standard the major AI crawlers have committed to. On crawlcheck.io, crawlers fetched it about as often as an ordinary page. Publishing it correctly is cheap; keep important facts in the pages themselves as well.

## Related findings

[2026-08-28 How to write an llms.txt and check it is actually served What goes in an llms.txt, which pages to list, whether generators help, and the fetch checks that prove crawlers get the file you](/guides/how-to-write-an-llms-txt-and-verify-it)[2026-10-03 AI crawler monitoring vs uptime monitors: what each one catches UptimeRobot, Visualping and CrawlCheck Watch compared: what each monitors, how often, price, and the AI crawler failures uptime ch](/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors)[2026-10-09 AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites 980 builder sites fetched as AI crawlers: llms.txt on nearly every Shopify and Wix site, Wix serves a lighter page, Squarespace le](/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [acxiom.com](/registry/acxiom.com) · [congress.gov](/registry/congress.gov) · [flippio.pl](/registry/flippio.pl) · [lhh.com](/registry/lhh.com) · [severadao.ai](/registry/severadao.ai) · [Place: Crestview](/registry/place/fl/crestview) | **Glossary:** [published crawler ranges](/glossary/published-crawler-ranges) · [Content-Digest](/glossary/content-digest)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/best-llms-txt-generators-and-validators-compared.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.