CrawlCheck

Guides · 2026-10-03 · By · 4 views

Cloudflare AI Crawl Control vs CrawlCheck: block vs measure

One controls AI crawler access at Cloudflare’s edge; the other measures, from outside, what each crawler is actually served. What each does, where each is weaker, and the routine that uses both.

Cloudflare AI Crawl Control is an edge control: on sites served through Cloudflare it shows AI crawler traffic and lets you allow or block each crawler, on all plans. CrawlCheck is a measurement: it fetches any domain as each AI crawler identifies itself and reports what each was actually served, including conflicts between robots.txt and firewall rules. Use Cloudflare to set the policy and a free CrawlCheck scan to confirm the result.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Cloudflare AI Crawl Control and CrawlCheck come up in the same conversations, and they are not substitutes. AI Crawl Control is a control: it runs at Cloudflare’s edge, in front of a site, and lets the owner see AI crawler traffic and allow, block or charge each crawler. CrawlCheck is a measurement: it fetches any site from outside, as each AI crawler identifies itself, and reports what each one was actually served. The control decides; the measurement confirms. Most sites on Cloudflare need both, because the commonest way a site loses AI visibility is a setting that does one thing while the owner believes it does another.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

Key points
  • Cloudflare AI Crawl Control is available on all Cloudflare plans and has four parts: manage AI crawlers, analyze AI traffic, track robots.txt, and pay per crawl (private beta), per Cloudflare’s documentation.
  • It acts only on sites served through Cloudflare. CrawlCheck measures any public domain.
  • On a Cloudflare site, bot management and WAF rules can still refuse a crawler that AI Crawl Control allows.
  • Across CrawlCheck’s scans, an answer engine’s crawler is refused while a browser is served on 5.8% of sites.
  • Use AI Crawl Control to set the policy and a CrawlCheck scan to confirm what each crawler is served.

What is Cloudflare AI Crawl Control? #

AI Crawl Control is Cloudflare’s set of tools for AI crawlers on a Cloudflare zone. Its documentation lists four parts. Manage AI crawlers lets the owner choose, crawler by crawler, how each may interact with the domain. Analyze AI traffic shows how AI crawlers are reaching the site’s pages. Track robots.txt checks the health of the robots.txt file and identifies crawlers that ignore its directives. Pay per crawl, in private beta, lets AI crawlers pay for access. Cloudflare describes the feature as available on all plans.

Because it runs on Cloudflare’s network, everything it does applies to real requests as they arrive. That is its strength: it sees the actual crawlers, from their actual addresses, and it can act on them before they reach the server.

What is CrawlCheck? #

CrawlCheck is an external measurement of how AI systems receive a site. A scan fetches the homepage and the machine files around it (robots.txt, sitemaps, llms.txt) as each crawler identifies itself, among them GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot, alongside a browser, and compares what each was served. It resolves robots.txt for 114 user-agents and checks the result against what each identity actually received. It works on any public domain, whatever the host or CDN, and it cannot block anything; it reports, and names the change that fixes each finding.

Side by side #

Cloudflare AI Crawl ControlCrawlCheck
TypeEdge control on your own Cloudflare zoneExternal measurement of any domain
SeesAI crawler requests that reach your site through CloudflareWhat each crawler identity is served when it fetches the site
Allow or block crawlersYes, per crawlerNo; it reports, and gives the exact fix
robots.txtTracks file health and which crawlers violate its directivesResolves robots.txt per crawler (114 user-agents) and compares it with what the edge served
Pay per crawlYes (private beta, per Cloudflare)No
Sites not on CloudflareNot coveredYes, any public domain
Content readable by AI (JavaScript-only text, pages that are mostly code, challenge pages)Not part of the featureYes
Real crawler traffic from real addressesYesNo; fetches from its own network under each crawler’s user-agent
PriceAvailable on all Cloudflare plansFirst scan free, then $3 a scan; Watch $29/mo for 3 domains

Sources: Cloudflare AI Crawl Control documentation; CrawlCheck pricing and scanner.

Why does a site on Cloudflare still refuse AI crawlers it allows? #

Because AI Crawl Control is not the only setting that decides a crawler’s request. Bot management, including Bot Fight Mode and bot scores, and WAF custom rules also act on requests, and the first layer that refuses wins, so robots.txt on the server is never read. The decisions live on different screens and none shows the combined outcome for, say, ClaudeBot on your homepage. A site in that state serves a browser normally and refuses an answer engine, which happens on 5.8% of scanned sites; the owner loads the page, sees it working and has no reason to look further. Is Cloudflare blocking AI crawlers on your site? covers each layer and how to find the one responsible.

Two views of the same site #

Inside view versus outside viewINSIDE: CLOUDFLARE AI CRAWL CONTROLSees and actsReal requests that reached your zoneWhich crawlers, how often, which pathsWhich crawlers ignore your robots.txtAllow, block, or charge per crawlerOnly sites served through CloudflareOnly after the fact, from real trafficOUTSIDE: CRAWLCHECKMeasures and comparesA fetch as each named crawlerStatus, size and text per identityrobots.txt resolved for 114 user-agentsChallenge pages, JavaScript-only textAny public domain, any hostCannot act; reports and names the fix
The two tools look at the same site from opposite sides. Neither view contains the other, which is why a policy set inside should be confirmed from outside.

The inside view is authoritative about real traffic: Cloudflare sees the genuine GPTBot, from OpenAI’s addresses, and records what happened to it. The outside view is authoritative about outcome per identity: it asks for the page the way each crawler asks, compares the answers side by side, and checks the content each was given. The inside view cannot tell you what a crawler that has not visited yet would get; the outside view cannot tell you how many times the real crawler came.

Which should you use? #

Which tool answers which questionWhat do you need to do?Block, allow or chargefor AI crawlers on aCloudflare siteProve thatChatGPT, Claude orPerplexity can read itKeep checkingafter changing a setting,or across many hostsAI Crawl Controlin the Cloudflare dashboardCrawlCheck scanfree, any domainBoth, in that orderchange in Cloudflare,then re-measure
Pick by the job, not the brand. Control lives at the edge; proof comes from a fetch made the way the crawler makes it.
  • You want to block or charge AI crawlers on a Cloudflare site: AI Crawl Control. CrawlCheck cannot block anything.
  • You want to know whether ChatGPT, Claude and Perplexity can read the site right now: a CrawlCheck scan, which works whether or not the site is on Cloudflare.
  • You changed a Cloudflare setting and want proof of the effect: both. Change it in Cloudflare, then re-scan to confirm what each crawler is served.
  • You manage sites on different hosts: CrawlCheck covers every host the same way; agency monitoring is $249 a month for 40 domains.

A routine that uses both #

The routine: set, measure, fix, re-measure1 Decideper crawler: train,search, user2 SetAI Crawl Controland robots.txt3 Measurescan as eachidentity4 Fixthe rule thatoverrides the policyre-measure after every Cloudflare change
A policy is only finished when a measurement confirms it. Watch repeats step 3 on a schedule and alerts when a crawler’s result changes.
  1. Decide a policy per crawler. Training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) are separate decisions. Blocking a training crawler does not remove a site from that company’s search results.
  2. Set it in AI Crawl Control and in robots.txt, so that the edge and the file say the same thing.
  3. Run a free CrawlCheck scan and read the result per identity: allowed by robots.txt, served by the edge, and given the same text as a browser.
  4. Fix the difference. It is usually a bot-management or WAF rule overriding the AI Crawl Control setting.
  5. Re-measure after every Cloudflare change. Watch repeats the scan on a schedule and alerts when a crawler’s result changes.

What does a mismatch look like in a report? #

A typical case reads like this: robots.txt allows ClaudeBot, AI Crawl Control is set to allow it, and the scan shows the browser identity receiving the homepage with status 200 and a full page of text while ClaudeBot receives a 403, or a 200 whose body is a short challenge page. The report puts the two responses next to each other, names the identity, and points to the layer most likely responsible. From there the fix is a Cloudflare setting, and the next scan confirms it.

Where CrawlCheck is weaker #

CrawlCheck sees what it is served when it fetches the site; it does not see the real AI crawlers’ traffic, and it cannot send requests from OpenAI’s or Anthropic’s own addresses. A rule keyed only to those companies’ IP ranges therefore shows up as a difference between identities rather than a confirmed block, and a rule that allows the published ranges while refusing everyone else using the crawler’s name will refuse CrawlCheck’s fetch too. Cloudflare’s dashboard, by contrast, sees every real request that reaches the zone and can act on it. For first-party traffic evidence and enforcement, use Cloudflare; for the per-crawler outcome from outside, on any host, use CrawlCheck.

More detail: how to block AI crawlers properly, how to verify GPTBot, ClaudeBot and PerplexityBot and why naming a crawler in robots.txt can drop your rules.

Share of scanned sites with each finding in this articleAnswer-engine crawler refused5.8%Share of scanned sites with eachfinding in this articleAnswer-engine crawler refused5.8%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What is the difference between Cloudflare AI Crawl Control and CrawlCheck?

AI Crawl Control is a Cloudflare feature that shows AI crawler traffic on your zone and lets you allow, block or charge each crawler. CrawlCheck is an external measurement that fetches any site as each AI crawler identifies itself and reports what each was served. One sets the policy; the other confirms its effect.

Is Cloudflare AI Crawl Control free?

Cloudflare's documentation says AI Crawl Control is available on all plans. Pay per crawl, which charges AI crawlers for access, is in private beta according to the same documentation.

Can Cloudflare block AI crawlers that I have allowed?

Yes. On a Cloudflare zone, bot management, Bot Fight Mode and WAF custom rules also act on requests and can refuse or challenge a crawler that robots.txt and AI Crawl Control allow. An external fetch as each crawler shows the combined result.

Does Cloudflare's managed robots.txt change my file?

Yes. With managed robots.txt on, Cloudflare puts its own rules before your file and serves both as one response, or creates the file if you have none. The file in your repository can say Allow while the live file says Disallow, so always test the live /robots.txt.

Does blocking GPTBot in Cloudflare remove my site from ChatGPT search?

Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search, so each needs its own decision in Cloudflare and in robots.txt.

Does CrawlCheck work for sites not on Cloudflare, and can it block crawlers?

It measures any public domain regardless of host or CDN, but it cannot block anything. It reports what each crawler was served and names the setting to change; blocking happens in robots.txt, the CDN or the firewall.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works