Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 4 views
Cloudflare AI Crawl Control vs CrawlCheck: block vs measure
One controls AI crawler access at Cloudflare’s edge; the other measures, from outside, what each crawler is actually served. What each does, where each is weaker, and the routine that uses both.
Cloudflare AI Crawl Control is an edge control: on sites served through Cloudflare it shows AI crawler traffic and lets you allow or block each crawler, on all plans. CrawlCheck is a measurement: it fetches any domain as each AI crawler identifies itself and reports what each was actually served, including conflicts between robots.txt and firewall rules. Use Cloudflare to set the policy and a free CrawlCheck scan to confirm the result.
Cloudflare AI Crawl Control and CrawlCheck come up in the same conversations, and they are not substitutes. AI Crawl Control is a control: it runs at Cloudflare’s edge, in front of a site, and lets the owner see AI crawler traffic and allow, block or charge each crawler. CrawlCheck is a measurement: it fetches any site from outside, as each AI crawler identifies itself, and reports what each one was actually served. The control decides; the measurement confirms. Most sites on Cloudflare need both, because the commonest way a site loses AI visibility is a setting that does one thing while the owner believes it does another.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.
- Cloudflare AI Crawl Control is available on all Cloudflare plans and has four parts: manage AI crawlers, analyze AI traffic, track robots.txt, and pay per crawl (private beta), per Cloudflare’s documentation.
- It acts only on sites served through Cloudflare. CrawlCheck measures any public domain.
- On a Cloudflare site, bot management and WAF rules can still refuse a crawler that AI Crawl Control allows.
- Across CrawlCheck’s scans, an answer engine’s crawler is refused while a browser is served on 5.8% of sites.
- Use AI Crawl Control to set the policy and a CrawlCheck scan to confirm what each crawler is served.
What is Cloudflare AI Crawl Control? #
AI Crawl Control is Cloudflare’s set of tools for AI crawlers on a Cloudflare zone. Its documentation lists four parts. Manage AI crawlers lets the owner choose, crawler by crawler, how each may interact with the domain. Analyze AI traffic shows how AI crawlers are reaching the site’s pages. Track robots.txt checks the health of the robots.txt file and identifies crawlers that ignore its directives. Pay per crawl, in private beta, lets AI crawlers pay for access. Cloudflare describes the feature as available on all plans.
Because it runs on Cloudflare’s network, everything it does applies to real requests as they arrive. That is its strength: it sees the actual crawlers, from their actual addresses, and it can act on them before they reach the server.
What is CrawlCheck? #
CrawlCheck is an external measurement of how AI systems receive a site. A scan fetches the homepage and the machine files around it (robots.txt, sitemaps, llms.txt) as each crawler identifies itself, among them GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot, alongside a browser, and compares what each was served. It resolves robots.txt for 114 user-agents and checks the result against what each identity actually received. It works on any public domain, whatever the host or CDN, and it cannot block anything; it reports, and names the change that fixes each finding.
Side by side #
| Cloudflare AI Crawl Control | CrawlCheck | |
|---|---|---|
| Type | Edge control on your own Cloudflare zone | External measurement of any domain |
| Sees | AI crawler requests that reach your site through Cloudflare | What each crawler identity is served when it fetches the site |
| Allow or block crawlers | Yes, per crawler | No; it reports, and gives the exact fix |
| robots.txt | Tracks file health and which crawlers violate its directives | Resolves robots.txt per crawler (114 user-agents) and compares it with what the edge served |
| Pay per crawl | Yes (private beta, per Cloudflare) | No |
| Sites not on Cloudflare | Not covered | Yes, any public domain |
| Content readable by AI (JavaScript-only text, pages that are mostly code, challenge pages) | Not part of the feature | Yes |
| Real crawler traffic from real addresses | Yes | No; fetches from its own network under each crawler’s user-agent |
| Price | Available on all Cloudflare plans | First scan free, then $3 a scan; Watch $29/mo for 3 domains |
Sources: Cloudflare AI Crawl Control documentation; CrawlCheck pricing and scanner.
Why does a site on Cloudflare still refuse AI crawlers it allows? #
Because AI Crawl Control is not the only setting that decides a crawler’s request. Bot management, including Bot Fight Mode and bot scores, and WAF custom rules also act on requests, and the first layer that refuses wins, so robots.txt on the server is never read. The decisions live on different screens and none shows the combined outcome for, say, ClaudeBot on your homepage. A site in that state serves a browser normally and refuses an answer engine, which happens on 5.8% of scanned sites; the owner loads the page, sees it working and has no reason to look further. Is Cloudflare blocking AI crawlers on your site? covers each layer and how to find the one responsible.
Two views of the same site #
The inside view is authoritative about real traffic: Cloudflare sees the genuine GPTBot, from OpenAI’s addresses, and records what happened to it. The outside view is authoritative about outcome per identity: it asks for the page the way each crawler asks, compares the answers side by side, and checks the content each was given. The inside view cannot tell you what a crawler that has not visited yet would get; the outside view cannot tell you how many times the real crawler came.
Which should you use? #
- You want to block or charge AI crawlers on a Cloudflare site: AI Crawl Control. CrawlCheck cannot block anything.
- You want to know whether ChatGPT, Claude and Perplexity can read the site right now: a CrawlCheck scan, which works whether or not the site is on Cloudflare.
- You changed a Cloudflare setting and want proof of the effect: both. Change it in Cloudflare, then re-scan to confirm what each crawler is served.
- You manage sites on different hosts: CrawlCheck covers every host the same way; agency monitoring is $249 a month for 40 domains.
A routine that uses both #
- Decide a policy per crawler. Training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) are separate decisions. Blocking a training crawler does not remove a site from that company’s search results.
- Set it in AI Crawl Control and in robots.txt, so that the edge and the file say the same thing.
- Run a free CrawlCheck scan and read the result per identity: allowed by robots.txt, served by the edge, and given the same text as a browser.
- Fix the difference. It is usually a bot-management or WAF rule overriding the AI Crawl Control setting.
- Re-measure after every Cloudflare change. Watch repeats the scan on a schedule and alerts when a crawler’s result changes.
What does a mismatch look like in a report? #
A typical case reads like this: robots.txt allows ClaudeBot, AI Crawl Control is set to allow it, and the scan shows the browser identity receiving the homepage with status 200 and a full page of text while ClaudeBot receives a 403, or a 200 whose body is a short challenge page. The report puts the two responses next to each other, names the identity, and points to the layer most likely responsible. From there the fix is a Cloudflare setting, and the next scan confirms it.
Where CrawlCheck is weaker #
CrawlCheck sees what it is served when it fetches the site; it does not see the real AI crawlers’ traffic, and it cannot send requests from OpenAI’s or Anthropic’s own addresses. A rule keyed only to those companies’ IP ranges therefore shows up as a difference between identities rather than a confirmed block, and a rule that allows the published ranges while refusing everyone else using the crawler’s name will refuse CrawlCheck’s fetch too. Cloudflare’s dashboard, by contrast, sees every real request that reaches the zone and can act on it. For first-party traffic evidence and enforcement, use Cloudflare; for the per-crawler outcome from outside, on any host, use CrawlCheck.
More detail: how to block AI crawlers properly, how to verify GPTBot, ClaudeBot and PerplexityBot and why naming a crawler in robots.txt can drop your rules.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is the difference between Cloudflare AI Crawl Control and CrawlCheck?
AI Crawl Control is a Cloudflare feature that shows AI crawler traffic on your zone and lets you allow, block or charge each crawler. CrawlCheck is an external measurement that fetches any site as each AI crawler identifies itself and reports what each was served. One sets the policy; the other confirms its effect.
Is Cloudflare AI Crawl Control free?
Cloudflare's documentation says AI Crawl Control is available on all plans. Pay per crawl, which charges AI crawlers for access, is in private beta according to the same documentation.
Can Cloudflare block AI crawlers that I have allowed?
Yes. On a Cloudflare zone, bot management, Bot Fight Mode and WAF custom rules also act on requests and can refuse or challenge a crawler that robots.txt and AI Crawl Control allow. An external fetch as each crawler shows the combined result.
Does Cloudflare's managed robots.txt change my file?
Yes. With managed robots.txt on, Cloudflare puts its own rules before your file and serves both as one response, or creates the file if you have none. The file in your repository can say Allow while the live file says Disallow, so always test the live /robots.txt.
Does blocking GPTBot in Cloudflare remove my site from ChatGPT search?
Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, so each needs its own decision in Cloudflare and in robots.txt.
Does CrawlCheck work for sites not on Cloudflare, and can it block crawlers?
It measures any public domain regardless of host or CDN, but it cannot block anything. It reports what each crawler was served and names the setting to change; blocking happens in robots.txt, the CDN or the firewall.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.