Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 1 view
Agent readiness checkers: Is It Agent Ready? vs CrawlCheck
Cloudflare’s Is It Agent Ready? and CrawlCheck compared: which agent standards each checks, declared versus working, why a score should skip checks that do not apply, and where each falls short.
Cloudflare’s free Is It Agent Ready? scores whether a site declares agent standards: robots.txt and Content Signals, Markdown negotiation, MCP Server Card, WebMCP, API Catalog, OAuth discovery and agent commerce protocols. CrawlCheck, also free, checks whether AI crawlers and agents are actually served the site and calls safe declared capabilities to confirm they work. Use CrawlCheck for access and working capabilities, Cloudflare’s checker for the newest standards, and ignore deductions for standards your kind of site does not need.
“Agent-ready” means a site that AI agents can discover, read and act on without guessing: it tells agents what they may do, serves content they can parse, and declares the tools or APIs they can call. Two free checkers score this today. Is It Agent Ready? is run by Cloudflare. CrawlCheck runs the other. They overlap less than the names suggest: one checks which agent standards a site declares, the other checks whether what is declared works and whether agents are served the site at all.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pages, linked where it appears, read on 3 October 2026. Where CrawlCheck does less, the tables say so.
What does each checker test? #
| Area | Is It Agent Ready? (Cloudflare) | CrawlCheck |
|---|---|---|
| Discovery | robots.txt, sitemap, Link response headers, DNS for AI Discovery (DNS-AID) | robots.txt resolved separately for each known AI crawler, sitemaps, llms.txt, and whether each is served to AI crawlers |
| Content | Markdown content negotiation | Whether each AI crawler is served the same text as a browser; JavaScript-only content; pages that are mostly code; challenge pages served as 200 |
| Bot access control | AI bot rules in robots.txt, Content Signals, Web Bot Auth | AI opt-out signals, shadowed robots.txt groups, refusals by firewall or CDN measured per identity; a free Web Bot Auth verifier |
| Protocols | MCP Server Card, Agent Skills, WebMCP, API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, ARD | Declared capabilities (API/OpenAPI, MCP, agent skills, A2A), with safe read-only ones called to confirm they work |
| Commerce | x402, MPP, UCP, ACP | Not checked |
| Output | A readiness score with AI-generated recommendations | Grade, every finding ranked, the exact fix, and a signed per-domain answer agents can read |
| Price | Free | First scan free, then $3 a scan; monitoring $29/mo for 3 domains |
Sources: isitagentready.com (© 2026 Cloudflare, Inc.), CrawlCheck capability registry and pricing.
Is a declared capability the same as a working one? #
No. A checker that looks for a file answers whether the file exists. An agent cares whether the thing it points to works. A declared endpoint that returns 404, or a tool that needs authentication it does not advertise, is a mismatch an agent will hit on first contact. The results across every measured site are public on the capability registry. If you run an MCP server, check it the way an agent would before you declare it.
Why should a score skip checks that do not apply? #
A checker that gives every site the same list measures the list, not the site. A public agent-readiness checker pointed at a Denver tree service returned 29 out of 100. Seven of the missing points sat in one category: no OAuth authorization server, no x402 payment endpoint, no MPP OpenAPI document, no UCP profile, no ACP document, no MCP card, no API catalog. The same output said “not a commerce site” next to four of those items, and counted them anyway.
A tree service with no payment-protocol endpoint is not defective. When you read any AEO or agent-readiness score, look at which deductions apply to your kind of business before acting on the number. A fair check that does not apply leaves the row out with the reason shown, and treats “could not tell” as unknown, not as a failure.
CrawlCheck broke the same rule once. A personal site that lists the trade brands its owner runs was graded as if it were the first of those businesses, and marked down for that company’s missing address, hours and phone on a page that was never about it. Pages that describe local businesses without being one are now left out of the local grade, with the reason stated, and the real local-business sites used as controls did not move. Every scored row was later reviewed against seven site types for the same reason.
How far each emerging surface has actually spread, read live from CrawlCheck’s scans when this page loads:
| Surface | Sites serving it | Share of 13965 scans |
|---|---|---|
| agents.md instructions written for an agent rather than a crawler | 3011 | 21.6% |
| Link headers registered relations only — describedby, service-desc | 6602 | 47.3% |
| Markdown negotiation a plain-text form of the page for a client that asks for one | 2781 | 19.9% |
Read from CrawlCheck’s own counters when the page loaded. None of these three is scored here, in either direction — they are reported with their evidence and left alone.
A surface that almost nobody serves describes where the standard is, not what a small business failed to do.
Why does the access layer come first? #
An agent that is refused never reaches your MCP card. A site can score well on declared standards and still fail all three of these, measured across every site CrawlCheck has scanned and read live.
An answer engine’s crawler is refused while a browser is served: 5.8%.
robots.txt groups named for AI crawlers silently drop the * rules: 14%.
An AI opt-out is already set: 1.8%.
How do you read the two results together? #
| Is It Agent Ready? | CrawlCheck | What it usually means |
|---|---|---|
| High score | AI crawlers refused or challenged | The declarations are there but agents cannot reach them; fix the firewall or CDN rule first |
| High score | Declared capability mismatches | A file points to an endpoint that is missing or needs undeclared authentication; fix the endpoint or the file |
| Low score | Clean access, no capabilities declared | Readable by agents; add declarations only for tools and APIs you actually run |
| Low score | Content needs JavaScript | Server-render the main content before adding anything else |
What is a minimal agent-ready setup? #
- Let crawlers in on purpose. A robots.txt that states a policy for the AI crawlers you care about, and a firewall that does not override it. Naming a crawler in its own group can cancel your * rules, so repeat them there.
- Serve the content in the HTML. The main text of each page should be in the response, not built afterwards by a script.
- Describe the business once. One coherent structured-data entity with the same name, address and phone as the page.
- Point to the pages that matter. A sitemap for everything and, optionally, an llms.txt for the few pages worth reading first.
- Declare only what works. An API description, an MCP server card or an agent skills file, each pointing at endpoints that answer as described.
Steps one to four apply to every site. Step five applies only to sites that offer something an agent can call; a brochure site with no API is not less agent-ready for leaving it out.
What does CrawlCheck call on your site? #
Confirming that a declared tool works means calling it, which is only acceptable when the call cannot change anything. CrawlCheck calls only public, read-only capabilities. Anything that writes or needs a credential is recorded as declared and left alone, and the report says which were called, so you can see exactly what was tested.
Where is CrawlCheck weaker? #
Cloudflare’s checker covers newer agent standards CrawlCheck does not test: DNS-AID, WebMCP, Auth.md, ARD, and the agent commerce protocols (x402, MPP, UCP, ACP). If you are implementing agent payments or the newest discovery records, use it. CrawlCheck does not generate AI recommendations; it names each finding and its fix from a fixed rulebook. Its page-level sections describe the home page, and its Core Web Vitals come from Google’s public field data, so they are absent when Google has no sample for the site.
Which should you run first? #
- Run a free CrawlCheck scan: confirm AI crawlers and agents are served the site and that declared capabilities work.
- Run Is It Agent Ready? for coverage of the newest discovery and commerce standards.
- Fix access problems before adding new declarations; a declaration behind a refusal is invisible.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is an agent-ready website?
One that AI agents can discover, read and act on: it states what agents may do (robots.txt, Content Signals), serves content they can parse, and declares tools or APIs they can call, such as an MCP server or an OpenAPI description, that actually work when called.
Who runs isitagentready.com?
Cloudflare. The site’s footer reads © 2026 Cloudflare, Inc. It is free and scores discoverability, content accessibility, bot access control, protocol discovery and agent commerce.
How is CrawlCheck different from Is It Agent Ready?
Is It Agent Ready? checks which agent standards a site declares, including newer ones such as DNS-AID, WebMCP and x402. CrawlCheck checks whether AI crawlers and agents are actually served the site, resolves robots.txt separately for each AI crawler, and calls safe read-only declared capabilities to confirm they work.
Why did my agent-readiness score drop for things my site does not need?
Some checkers apply every check to every site, so a service business loses points for having no payment protocol or OAuth server. Look at which deductions apply to your kind of business before acting on the number; a check that does not apply should be left out, not scored as a failure.
Does CrawlCheck check MCP servers?
CrawlCheck reads declared capabilities including MCP, API/OpenAPI, agent skills and A2A, and calls the public, read-only ones to confirm they answer as declared. Anything that writes or needs a credential is recorded but not called. Mismatches are listed on the public capability registry.
Are agent-readiness checkers free?
Yes. Is It Agent Ready? is free, and CrawlCheck's first scan is free with an email. CrawlCheck’s ongoing monitoring is a paid plan; current prices are on its pricing page.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.