Agent view · Guides · 2026-10-03
Agent readiness checkers: Is It Agent Ready? vs CrawlCheck
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 19,104 |
| words | 2,754 |
| sha-256 | 1770e7041de363074d67c1c363aba73a4ebe8025ee96c2e89b8b42c8891df92c |
| repr-digest | sha-256=:F3DnBB3jYwdNZ8HDY6unOk6+gCXulsLom4tCyIkd+Sw=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311489;expires=1795315089;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="wCVj8Gt6xaNBZq2flyD4-w";tag="crawlcheck-page" |
| signature | sig1=:AjE3inrivsDiHbLSvMBomUc3OZyKotvKnWnPGMTQEyevjmgjtxeAchKtDypKr14GE9wvSt3CDXCxkv3x3GhnBA==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 1 view # Agent readiness checkers: Is It Agent Ready? vs CrawlCheck Cloudflare’s Is It Agent Ready? and CrawlCheck compared: which agent standards each checks, declared versus working, why a score should skip checks that do not apply, and where each falls short. Cloudflare’s free Is It Agent Ready? scores whether a site declares agent standards: robots.txt and Content Signals, Markdown negotiation, MCP Server Card, WebMCP, API Catalog, OAuth discovery and agent commerce protocols. CrawlCheck, also free, checks whether AI crawlers and agents are actually served the site and calls safe declared capabilities to confirm they work. Use CrawlCheck for access and working capabilities, Cloudflare’s checker for the newest standards, and ignore deductions for standards your kind of site does not need. Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - What does each checker test? - Is a declared capability the same as a working one? - Why should a score skip checks that do not apply? - Why does the access layer come first? - How do you read the two results together? - What is a minimal agent-ready setup? - What does CrawlCheck call on your site? - Where is CrawlCheck weaker? - Which should you run first? “Agent-ready” means a site that AI agents can discover, read and act on without guessing: it tells agents what they may do, serves content they can parse, and declares the tools or APIs they can call. Two free checkers score this today. [Is It Agent Ready?](https://isitagentready.com/) is run by Cloudflare. CrawlCheck runs the other. They overlap less than the names suggest: one checks which agent standards a site declares, the other checks whether what is declared works and whether agents are served the site at all. **Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pages, linked where it appears, read on 3 October 2026. Where CrawlCheck does less, the tables say so. ## What does each checker test? # | Area | Is It Agent Ready? (Cloudflare) | CrawlCheck | | --- | --- | --- | | Discovery | [robots.txt](/glossary/robots-txt), sitemap, Link response headers, DNS for AI Discovery (DNS-AID) | robots.txt resolved separately for each known AI crawler, sitemaps, [llms.txt](/glossary/llms-txt), and whether each is served to AI crawlers | | Content | Markdown [content negotiation](/glossary/content-negotiation) | Whether each AI crawler is served the same text as a browser; JavaScript-only content; pages that are mostly code; challenge pages served as 200 | | Bot access control | AI bot rules in robots.txt, [Content Signals](/glossary/content-signals), [Web Bot Auth](/glossary/web-bot-auth) | [AI opt-out](/glossary/ai-opt-out) signals, shadowed robots.txt groups, refusals by firewall or [CDN](/glossary/cdn) measured per identity; a free Web Bot Auth verifier | | Protocols | [MCP](/glossary/mcp) Server Card, Agent Skills, [WebMCP](/glossary/webmcp), API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, ARD | Declared capabilities (API/OpenAPI, MCP, agent skills, [A2A](/glossary/a2a)), with safe read-only ones called to confirm they work | | Commerce | x402, MPP, UCP, ACP | Not checked | | Output | A readiness score with AI-generated recommendations | Grade, every finding ranked, the exact fix, and a signed per-domain answer agents can read | | Price | Free | First scan free, then $3 a scan; monitoring $29/mo for 3 domains | Sources: [isitagentready.com](https://isitagentready.com/) (© 2026 Cloudflare, Inc.), CrawlCheck [capability registry](/registry/capabilities) and [pricing](/pricing). ## Is a declared capability the same as a working one? # No. A checker that looks for a file answers whether the file exists. An agent cares whether the thing it points to works. A declared endpoint that returns 404, or a tool that needs authentication it does not advertise, is a mismatch an agent will hit on first contact. The results across every measured site are public on the [capability registry](/registry/capabilities). If you run an [MCP server](/glossary/mcp-server), [check it the way an agent would](/guides/how-to-check-an-mcp-server-before-an-agent-connects) before you declare it. ## Why should a score skip checks that do not apply? # A checker that gives every site the same list measures the list, not the site. A public agent-readiness checker pointed at a Denver tree service returned **29 out of 100**. Seven of the missing points sat in one category: no OAuth authorization server, no x402 payment endpoint, no MPP OpenAPI document, no UCP profile, no ACP document, no MCP card, no API catalog. The same output said “not a commerce site” next to four of those items, and counted them anyway. A tree service with no payment-protocol endpoint is not defective. When you read any [AEO](/glossary/aeo) or agent-readiness score, look at which deductions apply to your kind of business before acting on the number. A fair check that does not apply leaves the row out with the reason shown, and treats “could not tell” as unknown, not as a failure. CrawlCheck broke the same rule once. A personal site that lists the trade brands its owner runs was graded as if it were the first of those businesses, and marked down for that company’s missing address, hours and phone on a page that was never about it. Pages that describe local businesses without being one are now left out of the local grade, with the reason stated, and the real local-business sites used as controls did not move. [Every scored row was later reviewed against seven site types](/blog/we-read-every-scored-row-against-seven-sites-here-is-what-we-unscored) for the same reason. How far each emerging surface has actually spread, read live from CrawlCheck’s scans when this page loads: | Surface | Sites serving it | Share of 13966 scans | | --- | --- | --- | | agents.md instructions written for an agent rather than a crawler | 3011 | 21.6% | | Link headers registered relations only — describedby, service-desc | 6602 | 47.3% | | Markdown negotiation a plain-text form of the page for a client that asks for one | 2781 | 19.9% | Read from CrawlCheck’s own counters when the page loaded. None of these three is scored here, in either direction — they are reported with their evidence and left alone. A surface that almost nobody serves describes where the standard is, not what a small business failed to do. ## Why does the access layer come first? # An agent that is refused never reaches your MCP card. A site can score well on declared standards and still fail all three of these, measured across every site CrawlCheck has scanned and read live. An answer engine’s crawler is refused while a browser is served: **5.8%**. robots.txt groups named for AI crawlers silently drop the * rules: **14%**. An AI opt-out is already set: **1.8%**. ## How do you read the two results together? # | Is It Agent Ready? | CrawlCheck | What it usually means | | --- | --- | --- | | High score | AI crawlers refused or challenged | The declarations are there but agents cannot reach them; fix the firewall or CDN rule first | | High score | Declared capability mismatches | A file points to an endpoint that is missing or needs undeclared authentication; fix the endpoint or the file | | Low score | Clean access, no capabilities declared | Readable by agents; add declarations only for tools and APIs you actually run | | Low score | Content needs JavaScript | Server-render the main content before adding anything else | ## What is a minimal agent-ready setup? # - **Let crawlers in on purpose.** A robots.txt that states a policy for the AI crawlers you care about, and a firewall that does not override it. Naming a crawler in its own group [can cancel your * rules](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules), so repeat them there. - **Serve the content in the HTML.** The main text of each page should be in the response, not built afterwards by a script. - **Describe the business once.** One coherent structured-data entity with the same name, address and phone as the page. - **Point to the pages that matter.** A sitemap for everything and, optionally, an llms.txt for the few pages worth reading first. - **Declare only what works.** An API description, an MCP server card or an agent skills file, each pointing at endpoints that answer as described. Steps one to four apply to every site. Step five applies only to sites that offer something an agent can call; a brochure site with no API is not less agent-ready for leaving it out. ## What does CrawlCheck call on your site? # Confirming that a declared tool works means calling it, which is only acceptable when the call cannot change anything. CrawlCheck calls only public, read-only capabilities. Anything that writes or needs a credential is recorded as declared and left alone, and the report says which were called, so you can see exactly what was tested. ## Where is CrawlCheck weaker? # Cloudflare’s checker covers newer agent standards CrawlCheck does not test: DNS-AID, WebMCP, Auth.md, ARD, and the agent commerce protocols (x402, MPP, UCP, ACP). If you are implementing agent payments or the newest discovery records, use it. CrawlCheck does not generate AI recommendations; it names each finding and its fix from a fixed rulebook. Its page-level sections describe the home page, and its Core Web Vitals come from Google’s public field data, so they are absent when Google has no sample for the site. ## Which should you run first? # - Run a [free CrawlCheck scan](/): confirm AI crawlers and agents are served the site and that declared capabilities work. - Run Is It Agent Ready? for coverage of the newest discovery and commerce standards. - Fix access problems before adding new declarations; a declaration behind a refusal is invisible. Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fagent-readiness-checkers-compared-isitagentready-vs-crawlcheck%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fagent-readiness-checkers-compared-isitagentready-vs-crawlcheck%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Agent%20readiness%20checkers%3A%20Is%20It%20Agent%20Ready%3F%20vs%20CrawlCheck)[Email](mailto:?subject=Agent%20readiness%20checkers%3A%20Is%20It%20Agent%20Ready%3F%20vs%20CrawlCheck&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fagent-readiness-checkers-compared-isitagentready-vs-crawlcheck)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### What is an agent-ready website? One that AI agents can discover, read and act on: it states what agents may do (robots.txt, Content Signals), serves content they can parse, and declares tools or APIs they can call, such as an MCP server or an OpenAPI description, that actually work when called. ### Who runs isitagentready.com? Cloudflare. The site’s footer reads © 2026 Cloudflare, Inc. It is free and scores discoverability, content accessibility, bot access control, protocol discovery and agent commerce. ### How is CrawlCheck different from Is It Agent Ready? Is It Agent Ready? checks which agent standards a site declares, including newer ones such as DNS-AID, WebMCP and x402. CrawlCheck checks whether AI crawlers and agents are actually served the site, resolves robots.txt separately for each AI crawler, and calls safe read-only declared capabilities to confirm they work. ### Why did my agent-readiness score drop for things my site does not need? Some checkers apply every check to every site, so a service business loses points for having no payment protocol or OAuth server. Look at which deductions apply to your kind of business before acting on the number; a check that does not apply should be left out, not scored as a failure. ### Does CrawlCheck check MCP servers? CrawlCheck reads declared capabilities including MCP, API/OpenAPI, agent skills and A2A, and calls the public, read-only ones to confirm they answer as declared. Anything that writes or needs a credential is recorded but not called. Mismatches are listed on the public capability registry. ### Are agent-readiness checkers free? Yes. Is It Agent Ready? is free, and CrawlCheck's first scan is free with an email. CrawlCheck’s ongoing monitoring is a paid plan; current prices are on its pricing page. ## Related findings [2026-10-03 Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits Screaming Frog, Sitebulb, Semrush, Ahrefs and Lighthouse vs CrawlCheck: what each audit fetches, as whom, and which AI crawler pro](/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit)[2026-10-03 AI crawler monitoring vs uptime monitors: what each one catches UptimeRobot, Visualping and CrawlCheck Watch compared: what each monitors, how often, price, and the AI crawler failures uptime ch](/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors)[2026-10-03 Cloudflare AI Crawl Control vs CrawlCheck: block vs measure Cloudflare AI Crawl Control blocks or allows AI crawlers; CrawlCheck measures what each one is actually served. Features, limits,](/guides/cloudflare-ai-crawl-control-vs-crawlcheck) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [AI policy](/registry/aipolicy) · [bigbendchevrolet.com](/registry/bigbendchevrolet.com) · [docs.nginx.com](/registry/docs.nginx.com) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) | **Glossary:** [Conditional request](/glossary/conditional-request) · [Retrieval latency](/glossary/retrieval-latency) · [Information architecture](/glossary/information-architecture) · [Captions](/glossary/captions) · [incident switch](/glossary/incident-switch) | **Directory:** [Adept](/directory/adept) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.