Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 0 views
Agent readiness checkers: Is It Agent Ready? vs CrawlCheck
Cloudflare's Is It Agent Ready? and CrawlCheck compared: which agent standards each checks, declared versus working, the access layer underneath, and where each falls short.
Cloudflare's free Is It Agent Ready? scores whether a site declares agent standards: robots.txt and Content Signals, Markdown negotiation, MCP Server Card, WebMCP, API Catalog, OAuth discovery and agent commerce protocols. CrawlCheck, also free, checks whether AI crawlers and agents are actually served the site and calls safe declared capabilities to confirm they work. Use CrawlCheck for access and working capabilities, Cloudflare's checker for the newest standards.
“Agent-ready” means a site that AI agents can discover, read and act on without guessing: it tells agents what they may do, serves content they can parse, and declares the tools or APIs they can call. Two free checkers score this today. Is It Agent Ready? is run by Cloudflare. CrawlCheck runs the other. They overlap less than the names suggest: one checks which agent standards a site declares, the other checks whether what is declared actually works and whether agents are served the site at all.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.
What each one checks #
| Area | Is It Agent Ready? (Cloudflare) | CrawlCheck |
|---|---|---|
| Discovery | robots.txt, sitemap, Link response headers, DNS for AI Discovery (DNS-AID) | robots.txt resolved per crawler for 114 user-agents, sitemaps, llms.txt, and whether each is served to AI crawlers |
| Content | Markdown content negotiation | Whether each AI crawler is served the same text as a browser; JavaScript-only content; pages that are mostly code; challenge pages served as 200 |
| Bot access control | AI bot rules in robots.txt, Content Signals, Web Bot Auth | AI opt-out signals, shadowed robots.txt groups, refusals by firewall or CDN measured per identity; a free Web Bot Auth verifier |
| Protocols | MCP Server Card, Agent Skills, WebMCP, API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, ARD | Declared capabilities (API/OpenAPI, MCP, agent skills, A2A), with safe read-only ones called to confirm they work |
| Commerce | x402, MPP, UCP, ACP | Not checked |
| Output | A readiness score with AI-generated recommendations | Grade, every finding ranked, the exact fix, and a signed per-domain answer agents can read by API |
| Price | Free | Free scans; monitoring $29/mo for 3 domains |
Sources: isitagentready.com (© 2026 Cloudflare, Inc.), CrawlCheck capability registry and pricing.
Declared is not the same as working #
A checker that looks for a file answers whether the file exists. An agent cares whether the thing it points to works. CrawlCheck reads each capability a site declares and calls the safe, read-only ones to confirm they answer as declared. A declared endpoint that returns 404, or a tool that needs authentication it does not advertise, is a mismatch. The results across every measured site are public on the capability registry.
The access layer comes first #
An agent that is refused never reaches your MCP card. Across every site CrawlCheck has scanned, read live: an answer engine’s crawler is refused while a browser is served on 5.9%; robots.txt groups named for AI crawlers silently drop the * rules on 13.4%; an AI opt-out is already set on 5%. A site can score well on declared standards and still fail all three.
Reading the two results together #
| Is It Agent Ready? | CrawlCheck | What it usually means |
|---|---|---|
| High score | AI crawlers refused or challenged | The declarations are there but agents cannot reach them; fix the firewall or CDN rule first |
| High score | Declared capability mismatches | A file points to an endpoint that is missing or needs undeclared authentication; fix the endpoint or the file |
| Low score | Clean access, no capabilities declared | Readable by agents; add declarations only for tools and APIs you actually run |
| Low score | Content needs JavaScript | Server-render the main content before adding anything else |
A minimal agent-ready setup, in order #
- Let crawlers in on purpose. A robots.txt that states a policy for the AI crawlers you care about, and a firewall that does not override it.
- Serve the content in the HTML. The main text of each page should be in the response, not built afterwards by a script.
- Describe the business once. One coherent structured-data entity with the same name, address and phone as the page.
- Point to the pages that matter. A sitemap for everything and, optionally, an llms.txt for the few pages worth reading first.
- Declare only what works. An API description, an MCP server card or an agent skills file, each pointing at endpoints that answer as described.
Steps one to four apply to every site. Step five applies only to sites that offer something an agent can call; a brochure site with no API is not less agent-ready for leaving it out.
How CrawlCheck decides what is safe to call #
Confirming that a declared tool works means calling it, and calling a stranger’s tools is only acceptable when the call cannot change anything. CrawlCheck calls only public, read-only capabilities. Anything that writes or needs a credential is recorded as declared and left alone, and the result says which were called, so a site owner can see exactly what was tested.
Where CrawlCheck is weaker #
Cloudflare’s checker covers newer agent standards CrawlCheck does not test: DNS-AID, WebMCP, Auth.md, ARD, and the agent commerce protocols (x402, MPP, UCP, ACP). If you are implementing agent payments or the newest discovery records, use it. CrawlCheck also does not generate AI recommendations; it names each finding and its fix from a fixed rulebook.
Use both, in this order #
- Run a free CrawlCheck scan: confirm AI crawlers and agents are served the site and that declared capabilities work.
- Run Is It Agent Ready? for coverage of the newest discovery and commerce standards.
- Fix access problems before adding new declarations; a declaration behind a refusal is invisible.
Agents can also read CrawlCheck’s answer about any domain directly: see the resolve protocol.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
What is an agent-ready website?
One that AI agents can discover, read and act on: it states what agents may do (robots.txt, Content Signals), serves content they can parse, and declares tools or APIs they can call, such as an MCP server or an OpenAPI description, that actually work when called.
Who runs isitagentready.com?
Cloudflare. The site's footer reads © 2026 Cloudflare, Inc. It is free and scores discoverability, content accessibility, bot access control, protocol discovery and agent commerce.
How is CrawlCheck different from Is It Agent Ready?
Is It Agent Ready? checks which agent standards a site declares, including newer ones such as DNS-AID, WebMCP and x402. CrawlCheck checks whether AI crawlers and agents are actually served the site, resolves robots.txt for 114 user-agents, and calls safe read-only declared capabilities to confirm they work.
Does CrawlCheck check MCP servers?
CrawlCheck reads declared capabilities including MCP, API/OpenAPI, agent skills and A2A, and calls the public, read-only ones to confirm they answer as declared. Mismatches are reported and listed on the public capability registry.
Are agent-readiness checkers free?
Yes. Is It Agent Ready? is free, and CrawlCheck scans are free with no account. CrawlCheck's ongoing monitoring is $29 a month for up to three domains.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.