Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 3 views
Best llms.txt generators and validators compared
Yoast, Mintlify, Firecrawl and CrawlCheck compared on writing llms.txt, keeping it current, and checking that AI crawlers actually receive it.
Yoast SEO generates and refreshes llms.txt automatically for WordPress and Shopify, free. Mintlify hosts llms.txt and llms-full.txt for documentation sites. Firecrawl's free generator builds llms.txt and llms-full.txt for any site, and CrawlCheck drafts one from your declared pages. None of the generators checks the published file. CrawlCheck's free scan does: it fetches the file as AI crawlers do and reports whether they get text, a current copy and the same file on every host.
An llms.txt is a Markdown file at the root of a site that points AI tools to its most useful pages. The proposal at llmstxt.org (by Jeremy Howard) requires only an H1 with the site’s name. Writing one is now easy: several tools do it automatically. What none of the generators checks is whether the published file actually reaches AI crawlers as text. The format itself is covered in how to write an llms.txt and verify it; this page compares the tools.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every claim about another company comes from that company’s own pricing or documentation page, linked below, read on 3 October 2026. Where CrawlCheck does less than a competitor, the table says so.
Which tools generate llms.txt, and which check it? #
| Tool | What it does | Keeps it updated? | Checks it is served to crawlers? | Price |
|---|---|---|---|---|
| CrawlCheck | Drafts an llms.txt from your homepage and declared pages; audits the live file | No (draft only) | Yes: fetches /llms.txt on apex and www as AI crawlers do and reports status, content type, freshness and whether the pages it lists resolve | Free |
| Firecrawl llms.txt generator | Crawls a site and generates llms.txt and llms-full.txt (full page content) | No; regenerate on demand | No | Free; a free API key removes limits |
| Yoast SEO | Generates llms.txt for WordPress and Shopify sites, automatic or manual mode | Yes, automatically | No | Included free in Yoast |
| Mintlify | Hosts llms.txt and llms-full.txt for documentation sites | Yes, automatically | No | Part of Mintlify docs hosting |
Sources: Firecrawl, Yoast, Mintlify, and the free CrawlCheck scan.
What does a generator not check? #
A file that exists in your CMS can still fail to reach a crawler, and every one of these failures looks fine in a browser. The same pattern on robots.txt, a challenge page answered with 200, is in a file that answered 200 and could not be read; the cache cases are in the machine-file caching guide.
| Failure | What a crawler gets | How to see it |
|---|---|---|
| Page template at /llms.txt | HTML or a login wall, with status 200 | curl -sI: content type is text/html |
| Stale cache | The copy from before your last edit | Request with and without a cache-busting query; compare |
| Apex and www differ | Whichever file the host it started on serves | Fetch both hosts; compare |
| Bot rule | A refusal the browser never sees | Fetch as each crawler, not as a browser |
How common are missing and stale files? #
Two live figures from CrawlCheck scans. Both count sites people chose to scan with CrawlCheck, not a fixed list of top domains, so they answer a different question from a census of the web and the two shares should not be compared directly. Share of scanned sites with no llms.txt at all: 34.1%.
Share of scanned sites where a cache serves crawlers an out-of-date machine file: 0.4%.
Which tool should you use? #
- WordPress or Shopify on Yoast: turn on Yoast’s llms.txt; it updates itself.
- Documentation on Mintlify: it is already there.
- Any other site: generate a draft with Firecrawl’s generator or CrawlCheck’s free drafter, cut it to the pages that actually answer customers’ questions, and publish it.
- Every site, after publishing: run a free scan to confirm AI crawlers receive the file as text.
The first check takes one command: curl -sI https://example.com/llms.txt, then the same for the www host, reading the status line and the content-type header. The full post-publish checklist is in the writing guide.
Where is CrawlCheck weaker? #
CrawlCheck’s drafter works from your homepage and declared pages and does not keep the file updated; Yoast and Mintlify do that automatically. Firecrawl also produces llms-full.txt with full page content, which CrawlCheck does not. CrawlCheck’s strength is the check after publishing.
Is llms.txt worth publishing at all? #
It is a proposal, not a standard the major AI crawlers have committed to read. On our own site, where every page links it, sixteen crawler names fetched it 8 times in two weeks, about as often as an ordinary blog post (the measurement). Treat it as low-cost and low-certainty: worth publishing correctly, not worth publishing broken, and never the only place an important fact appears. The wider evidence is in does llms.txt actually work?
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is the best llms.txt generator?
For WordPress or Shopify sites on Yoast SEO, Yoast's built-in llms.txt is free and updates automatically. Mintlify documentation sites get llms.txt and llms-full.txt automatically. For any other site, Firecrawl's free generator and CrawlCheck's free drafter create a starting file. Whatever you use, confirm the live file is served to AI crawlers as text.
Is there a free llms.txt validator?
Yes. CrawlCheck's free llms.txt audit checks /llms.txt on the apex and www hosts as AI crawlers receive it: status, content type, cache freshness and whether the pages it cites resolve. A file that answers 200 with HTML or a login wall is not a working llms.txt.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short Markdown index with links to a site's key pages. llms-full.txt, produced by tools such as Firecrawl and Mintlify, includes the full content of each page in one file.
Does Yoast create llms.txt?
Yes. Yoast's llms.txt feature is free in Yoast SEO, Premium, WooCommerce SEO, AI+ and Yoast SEO for Shopify, and generates and refreshes the file automatically, with a manual mode for choosing content.
Why does my llms.txt show a web page instead of text?
Usually because the CMS routes unknown paths to a page template, or a login or bot challenge sits in front of the file, and both can answer with status 200. Check the content-type header: a working llms.txt is served as text/plain or text/markdown, not text/html.
Do AI crawlers read llms.txt?
It is a proposal rather than a standard the major AI crawlers have committed to. On crawlcheck.io, crawlers fetched it about as often as an ordinary page. Publishing it correctly is cheap; keep important facts in the pages themselves as well.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.