CrawlCheck

Agent view · Guides · 2026-10-03

Cloudflare AI Crawl Control vs CrawlCheck: block vs measure

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/cloudflare-ai-crawl-control-vs-crawlcheck: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/cloudflare-ai-crawl-control-vs-crawlcheck>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes19,479
words2,831
sha-2567e367fe53dfa1f731f4b28223887ca5e916af397fdc41ba20f91ba83e67492cb
repr-digestsha-256=:fjZ/5T36H3MfSygiOIfKXpFq85f9xBuiD5G6g+Z0kss=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311484;expires=1795315084;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="NXr889mhSp3UBun23J8koQ";tag="crawlcheck-page"
signaturesig1=:e1I9L5BHfiQs6xE9DTeA3Zuo+84hPgFnrGjdS60L0SRrHdIkBnw8+2VgRXoqU73GnLOJMSXMcVOT7RkNZ/ezDA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 4 views

# Cloudflare AI Crawl Control vs CrawlCheck: block vs measure

One controls AI crawler access at Cloudflare’s edge; the other measures, from outside, what each crawler is actually served. What each does, where each is weaker, and the routine that uses both.

Cloudflare AI Crawl Control is an edge control: on sites served through Cloudflare it shows AI crawler traffic and lets you allow or block each crawler, on all plans. CrawlCheck is a measurement: it fetches any domain as each AI crawler identifies itself and reports what each was actually served, including conflicts between robots.txt and firewall rules. Use Cloudflare to set the policy and a free CrawlCheck scan to confirm the result.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What is Cloudflare AI Crawl Control?
- What is CrawlCheck?
- Side by side
- Why does a site on Cloudflare still refuse AI crawlers it allows?
- Two views of the same site
- Which should you use?
- A routine that uses both
- What does a mismatch look like in a report?
- Where CrawlCheck is weaker

Cloudflare AI Crawl Control and CrawlCheck come up in the same conversations, and they are not substitutes. **AI Crawl Control is a control**: it runs at Cloudflare’s edge, in front of a site, and lets the owner see AI crawler traffic and allow, block or charge each crawler. **CrawlCheck is a measurement**: it fetches any site from outside, as each AI crawler identifies itself, and reports what each one was actually served. The control decides; the measurement confirms. Most sites on Cloudflare need both, because the commonest way a site loses AI visibility is a setting that does one thing while the owner believes it does another.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

**Key points**

- Cloudflare AI Crawl Control is available on all Cloudflare plans and has four parts: manage AI crawlers, analyze AI traffic, track [robots.txt](/glossary/robots-txt), and pay per crawl (private beta), per Cloudflare’s documentation.
- It acts only on sites served through Cloudflare. CrawlCheck measures any public domain.
- On a Cloudflare site, bot management and [WAF](/glossary/waf) rules can still refuse a crawler that AI Crawl Control allows.
- Across CrawlCheck’s scans, an [answer engine](/glossary/answer-engine)’s crawler is refused while a browser is served on 5.8% of sites.
- Use AI Crawl Control to set the policy and a CrawlCheck scan to confirm what each crawler is served.

## What is Cloudflare AI Crawl Control? #

AI Crawl Control is Cloudflare’s set of tools for AI crawlers on a Cloudflare zone. Its [documentation](https://developers.cloudflare.com/ai-crawl-control/) lists four parts. **Manage AI crawlers** lets the owner choose, crawler by crawler, how each may interact with the domain. **Analyze AI traffic** shows how AI crawlers are reaching the site’s pages. **Track robots.txt** checks the health of the robots.txt file and identifies crawlers that ignore its directives. **Pay per crawl**, in private beta, lets AI crawlers pay for access. Cloudflare describes the feature as available on all plans.

Because it runs on Cloudflare’s network, everything it does applies to real requests as they arrive. That is its strength: it sees the actual crawlers, from their actual addresses, and it can act on them before they reach the server.

## What is CrawlCheck? #

CrawlCheck is an external measurement of how AI systems receive a site. A scan fetches the homepage and the machine files around it (robots.txt, sitemaps, [llms.txt](/glossary/llms-txt)) as each crawler identifies itself, among them GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot, alongside a browser, and compares what each was served. It resolves robots.txt for 114 user-agents and checks the result against what each identity actually received. It works on any public domain, whatever the host or [CDN](/glossary/cdn), and it cannot block anything; it reports, and names the change that fixes each finding.

## Side by side #

| | Cloudflare AI Crawl Control | CrawlCheck |
| --- | --- | --- |
| Type | Edge control on your own Cloudflare zone | External measurement of any domain |
| Sees | AI crawler requests that reach your site through Cloudflare | What each crawler identity is served when it fetches the site |
| Allow or block crawlers | Yes, per crawler | No; it reports, and gives the exact fix |
| robots.txt | Tracks file health and which crawlers violate its directives | Resolves robots.txt per crawler (114 user-agents) and compares it with what the edge served |
| Pay per crawl | Yes (private beta, per Cloudflare) | No |
| Sites not on Cloudflare | Not covered | Yes, any public domain |
| Content readable by AI (JavaScript-only text, pages that are mostly code, challenge pages) | Not part of the feature | Yes |
| Real crawler traffic from real addresses | Yes | No; fetches from its own network under each crawler’s user-agent |
| Price | Available on all Cloudflare plans | First scan free, then $3 a scan; Watch $29/mo for 3 domains |

Sources: [Cloudflare AI Crawl Control documentation](https://developers.cloudflare.com/ai-crawl-control/); CrawlCheck [pricing](/pricing) and [scanner](/).

## Why does a site on Cloudflare still refuse AI crawlers it allows? #

Because AI Crawl Control is not the only setting that decides a crawler’s request. Bot management, including Bot Fight Mode and bot scores, and WAF custom rules also act on requests, and the first layer that refuses wins, so robots.txt on the server is never read. The decisions live on different screens and none shows the combined outcome for, say, ClaudeBot on your homepage. A site in that state serves a browser normally and refuses an answer engine, which happens on 5.8% of scanned sites; the owner loads the page, sees it working and has no reason to look further. [Is Cloudflare blocking AI crawlers on your site?](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site) covers each layer and how to find the one responsible.

## Two views of the same site #

The two tools look at the same site from opposite sides. Neither view contains the other, which is why a policy set inside should be confirmed from outside.
The inside view is authoritative about real traffic: Cloudflare sees the genuine GPTBot, from OpenAI’s addresses, and records what happened to it. The outside view is authoritative about outcome per identity: it asks for the page the way each crawler asks, compares the answers side by side, and checks the content each was given. The inside view cannot tell you what a crawler that has not visited yet would get; the outside view cannot tell you how many times the real crawler came.

## Which should you use? #

Pick by the job, not the brand. Control lives at the edge; proof comes from a fetch made the way the crawler makes it.

- **You want to block or charge AI crawlers on a Cloudflare site:** AI Crawl Control. CrawlCheck cannot block anything.
- **You want to know whether ChatGPT, Claude and Perplexity can read the site right now:** a CrawlCheck scan, which works whether or not the site is on Cloudflare.
- **You changed a Cloudflare setting and want proof of the effect:** both. Change it in Cloudflare, then re-scan to confirm what each crawler is served.
- **You manage sites on different hosts:** CrawlCheck covers every host the same way; agency monitoring is $249 a month for 40 domains.

## A routine that uses both #

A policy is only finished when a measurement confirms it. Watch repeats step 3 on a schedule and alerts when a crawler’s result changes.

- **Decide a policy per crawler.** Training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) are separate decisions. Blocking a training crawler does not remove a site from that company’s search results.
- **Set it in AI Crawl Control and in robots.txt,** so that the edge and the file say the same thing.
- **Run a free CrawlCheck scan** and read the result per identity: allowed by robots.txt, served by the edge, and given the same text as a browser.
- **Fix the difference.** It is usually a bot-management or WAF rule overriding the AI Crawl Control setting.
- **Re-measure after every Cloudflare change.** Watch repeats the scan on a schedule and alerts when a crawler’s result changes.

## What does a mismatch look like in a report? #

A typical case reads like this: robots.txt allows ClaudeBot, AI Crawl Control is set to allow it, and the scan shows the browser identity receiving the homepage with status 200 and a full page of text while ClaudeBot receives a 403, or a 200 whose body is a short challenge page. The report puts the two responses next to each other, names the identity, and points to the layer most likely responsible. From there the fix is a Cloudflare setting, and the next scan confirms it.

## Where CrawlCheck is weaker #

CrawlCheck sees what it is served when it fetches the site; it does not see the real AI crawlers’ traffic, and it cannot send requests from OpenAI’s or Anthropic’s own addresses. A rule keyed only to those companies’ IP ranges therefore shows up as a difference between identities rather than a confirmed block, and a rule that allows the published ranges while refusing everyone else using the crawler’s name will refuse CrawlCheck’s fetch too. Cloudflare’s dashboard, by contrast, sees every real request that reaches the zone and can act on it. For first-party traffic evidence and enforcement, use Cloudflare; for the per-crawler outcome from outside, on any host, use CrawlCheck.

More detail: [how to block AI crawlers properly](/guides/how-to-block-ai-crawlers), [how to verify GPTBot, ClaudeBot and PerplexityBot](/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared) and [why naming a crawler in robots.txt can drop your rules](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules).
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcloudflare-ai-crawl-control-vs-crawlcheck%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcloudflare-ai-crawl-control-vs-crawlcheck%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Cloudflare%20AI%20Crawl%20Control%20vs%20CrawlCheck%3A%20block%20vs%20measure)[Email](mailto:?subject=Cloudflare%20AI%20Crawl%20Control%20vs%20CrawlCheck%3A%20block%20vs%20measure&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcloudflare-ai-crawl-control-vs-crawlcheck)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is the difference between Cloudflare AI Crawl Control and CrawlCheck?

AI Crawl Control is a Cloudflare feature that shows AI crawler traffic on your zone and lets you allow, block or charge each crawler. CrawlCheck is an external measurement that fetches any site as each AI crawler identifies itself and reports what each was served. One sets the policy; the other confirms its effect.

### Is Cloudflare AI Crawl Control free?

Cloudflare's documentation says AI Crawl Control is available on all plans. Pay per crawl, which charges AI crawlers for access, is in private beta according to the same documentation.

### Can Cloudflare block AI crawlers that I have allowed?

Yes. On a Cloudflare zone, bot management, Bot Fight Mode and WAF custom rules also act on requests and can refuse or challenge a crawler that robots.txt and AI Crawl Control allow. An external fetch as each crawler shows the combined result.

### Does Cloudflare's managed robots.txt change my file?

Yes. With managed robots.txt on, Cloudflare puts its own rules before your file and serves both as one response, or creates the file if you have none. The file in your repository can say Allow while the live file says Disallow, so always test the live /robots.txt.

### Does blocking GPTBot in Cloudflare remove my site from ChatGPT search?

Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, so each needs its own decision in Cloudflare and in robots.txt.

### Does CrawlCheck work for sites not on Cloudflare, and can it block crawlers?

It measures any public domain regardless of host or CDN, but it cannot block anything. It reports what each crawler was served and names the setting to change; blocking happens in robots.txt, the CDN or the firewall.

## Related findings

[2026-10-26 Google-Extended and Applebot-Extended: block AI training Google-Extended and Applebot-Extended are tokens, not crawlers: they opt a site out of AI training and keep it in search. The line](/guides/google-extended-and-applebot-extended-opt-out-of-ai-training-keep-search)[2026-08-28 How to write an llms.txt and check it is actually served What goes in an llms.txt, which pages to list, whether generators help, and the fetch checks that prove crawlers get the file you](/guides/how-to-write-an-llms-txt-and-verify-it)[2026-10-03 Agent readiness checkers: Is It Agent Ready? vs CrawlCheck Is It Agent Ready? vs CrawlCheck: agent standards checked, declared vs working capabilities, AI crawler access, scores that skip w](/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [MCP tool changes](/research/mcp-tool-changes) · [Conduct: Google safety](/agents/conduct/google-safety) | **Blog:** [Our mirror relayed real crawlers and inflated forgery counts](/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts) | **Glossary:** [finding age](/glossary/finding-age) · [Deduplication](/glossary/deduplication) · [Commercial investigation](/glossary/commercial-investigation) · [Accessible description](/glossary/accessible-description) · [agent task canary](/glossary/agent-task-canary)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/cloudflare-ai-crawl-control-vs-crawlcheck.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.