CrawlCheck

Guides · 2026-10-03 · By · 3 views

Best llms.txt generators and validators compared

Yoast, Mintlify, Firecrawl and CrawlCheck compared on writing llms.txt, keeping it current, and checking that AI crawlers actually receive it.

Yoast SEO generates and refreshes llms.txt automatically for WordPress and Shopify, free. Mintlify hosts llms.txt and llms-full.txt for documentation sites. Firecrawl's free generator builds llms.txt and llms-full.txt for any site, and CrawlCheck drafts one from your declared pages. None of the generators checks the published file. CrawlCheck's free scan does: it fetches the file as AI crawlers do and reports whether they get text, a current copy and the same file on every host.

Check your own site free

About a minute. First scan free with your email, nothing installed.

An llms.txt is a Markdown file at the root of a site that points AI tools to its most useful pages. The proposal at llmstxt.org (by Jeremy Howard) requires only an H1 with the site’s name. Writing one is now easy: several tools do it automatically. What none of the generators checks is whether the published file actually reaches AI crawlers as text. The format itself is covered in how to write an llms.txt and verify it; this page compares the tools.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every claim about another company comes from that company’s own pricing or documentation page, linked below, read on 3 October 2026. Where CrawlCheck does less than a competitor, the table says so.

Which tools generate llms.txt, and which check it? #

ToolWhat it doesKeeps it updated?Checks it is served to crawlers?Price
CrawlCheckDrafts an llms.txt from your homepage and declared pages; audits the live fileNo (draft only)Yes: fetches /llms.txt on apex and www as AI crawlers do and reports status, content type, freshness and whether the pages it lists resolveFree
Firecrawl llms.txt generatorCrawls a site and generates llms.txt and llms-full.txt (full page content)No; regenerate on demandNoFree; a free API key removes limits
Yoast SEOGenerates llms.txt for WordPress and Shopify sites, automatic or manual modeYes, automaticallyNoIncluded free in Yoast
MintlifyHosts llms.txt and llms-full.txt for documentation sitesYes, automaticallyNoPart of Mintlify docs hosting

Sources: Firecrawl, Yoast, Mintlify, and the free CrawlCheck scan.

What does a generator not check? #

1. Generatea tool writes the file2. Publishat /llms.txt3. Servedwhat a crawler receivesGenerators stop at step 2. The failures happen between 2 and 3:HTML page or login wall answered with 200an old copy kept by a cache after you edita different file on the apex and the www hosta bot rule refusing AI crawlers the file a browser getsA browser check passes all four. Only a fetch as each crawler shows them.
Generating the file and delivering it are separate steps; no generator in this comparison checks the second.

A file that exists in your CMS can still fail to reach a crawler, and every one of these failures looks fine in a browser. The same pattern on robots.txt, a challenge page answered with 200, is in a file that answered 200 and could not be read; the cache cases are in the machine-file caching guide.

FailureWhat a crawler getsHow to see it
Page template at /llms.txtHTML or a login wall, with status 200curl -sI: content type is text/html
Stale cacheThe copy from before your last editRequest with and without a cache-busting query; compare
Apex and www differWhichever file the host it started on servesFetch both hosts; compare
Bot ruleA refusal the browser never seesFetch as each crawler, not as a browser

How common are missing and stale files? #

Two live figures from CrawlCheck scans. Both count sites people chose to scan with CrawlCheck, not a fixed list of top domains, so they answer a different question from a census of the web and the two shares should not be compared directly. Share of scanned sites with no llms.txt at all: 34.1%.

Share of scanned sites where a cache serves crawlers an out-of-date machine file: 0.4%.

Which tool should you use? #

  • WordPress or Shopify on Yoast: turn on Yoast’s llms.txt; it updates itself.
  • Documentation on Mintlify: it is already there.
  • Any other site: generate a draft with Firecrawl’s generator or CrawlCheck’s free drafter, cut it to the pages that actually answer customers’ questions, and publish it.
  • Every site, after publishing: run a free scan to confirm AI crawlers receive the file as text.

The first check takes one command: curl -sI https://example.com/llms.txt, then the same for the www host, reading the status line and the content-type header. The full post-publish checklist is in the writing guide.

Where is CrawlCheck weaker? #

CrawlCheck’s drafter works from your homepage and declared pages and does not keep the file updated; Yoast and Mintlify do that automatically. Firecrawl also produces llms-full.txt with full page content, which CrawlCheck does not. CrawlCheck’s strength is the check after publishing.

Is llms.txt worth publishing at all? #

It is a proposal, not a standard the major AI crawlers have committed to read. On our own site, where every page links it, sixteen crawler names fetched it 8 times in two weeks, about as often as an ordinary blog post (the measurement). Treat it as low-cost and low-certainty: worth publishing correctly, not worth publishing broken, and never the only place an important fact appears. The wider evidence is in does llms.txt actually work?

Share of scanned sites with each finding in this articleNo llms.txt34.1%Stale cached robots.txt or llms.txt0.4%Share of scanned sites with eachfinding in this articleNo llms.txt34.1%Stale cached robots.txt or llms.txt0.4%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What is the best llms.txt generator?

For WordPress or Shopify sites on Yoast SEO, Yoast's built-in llms.txt is free and updates automatically. Mintlify documentation sites get llms.txt and llms-full.txt automatically. For any other site, Firecrawl's free generator and CrawlCheck's free drafter create a starting file. Whatever you use, confirm the live file is served to AI crawlers as text.

Is there a free llms.txt validator?

Yes. CrawlCheck's free llms.txt audit checks /llms.txt on the apex and www hosts as AI crawlers receive it: status, content type, cache freshness and whether the pages it cites resolve. A file that answers 200 with HTML or a login wall is not a working llms.txt.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short Markdown index with links to a site's key pages. llms-full.txt, produced by tools such as Firecrawl and Mintlify, includes the full content of each page in one file.

Does Yoast create llms.txt?

Yes. Yoast's llms.txt feature is free in Yoast SEO, Premium, WooCommerce SEO, AI+ and Yoast SEO for Shopify, and generates and refreshes the file automatically, with a manual mode for choosing content.

Why does my llms.txt show a web page instead of text?

Usually because the CMS routes unknown paths to a page template, or a login or bot challenge sits in front of the file, and both can answer with status 200. Check the content-type header: a working llms.txt is served as text/plain or text/markdown, not text/html.

Do AI crawlers read llms.txt?

It is a proposal rather than a standard the major AI crawlers have committed to. On crawlcheck.io, crawlers fetched it about as often as an ordinary page. Publishing it correctly is cheap; keep important facts in the pages themselves as well.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works