[AI crawlers](/ai-crawlers) · SEO and research crawlers

# `Cloudflare-AI-Search-External`

`Cloudflare-AI-Search-External` is the tool or developer fetch run by Cloudflare: Cloudflare AI Search External. Verify the HTTP Message Signature against the operator’s public key directory: https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory.

| | |
| --- | --- |
| User-agent token | `Cloudflare-AI-Search-External` |
| Operator | Cloudflare |
| What it does | Cloudflare AI Search External |
| Kind | tool or developer fetch |
| Can a request be verified? | Yes — web bot auth |
| Verification source | https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory |
| Ready-made robots.txt | [Block training and corpus crawlers](/robots/block-ai-training.txt) · [Block every AI crawler](/robots/block-all-ai.txt) · [Block SEO and ad crawlers](/robots/block-seo-tools.txt) · [server rules](/robots-policies) · [this page as JSON](/ai-crawlers/cloudflare-ai-search-external.json) |
| In a CrawlCheck scan | Read from robots.txt; reported, never scored |

## Allow or block it in robots.txt

```
User-agent: Cloudflare-AI-Search-External
Disallow: /
```

```
User-agent: Cloudflare-AI-Search-External
Allow: /
```

Add a group that names it: "User-agent: Cloudflare-AI-Search-External" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Cloudflare-AI-Search-External, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

## What blocking it changes

No. It is a third-party tool's crawler. Blocking it changes what that tool knows about your site, not anyone's AI answers.

## Other agents from Cloudflare

- [CloudflareBrowserRenderingCrawler](/ai-crawlers/cloudflarebrowserrenderingcrawler) — Cloudflare Browser Run /crawl
- [Cloudflare-AutoRAG](/ai-crawlers/cloudflare-autorag) — Cloudflare AutoRAG

See what `Cloudflare-AI-Search-External` and every other named agent receive from your own site: [run a free scan](/#scanForm). The full catalogue is on [AI crawlers](/ai-crawlers) and as JSON at [/ai-crawlers.json](/ai-crawlers.json).

## Questions about Cloudflare-AI-Search-External

#### Q How do I block Cloudflare-AI-Search-External in robots.txt?

Add a group that names it: "User-agent: Cloudflare-AI-Search-External" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Cloudflare-AI-Search-External, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

#### Q How can I tell if a request from Cloudflare-AI-Search-External is real?

Verify the HTTP Message Signature against the operator’s public key directory: https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory.

#### Q Does blocking Cloudflare-AI-Search-External affect whether my site appears in AI answers?

No. It is a third-party tool's crawler. Blocking it changes what that tool knows about your site, not anyone's AI answers.

#### Q Does CrawlCheck test my site as Cloudflare-AI-Search-External?

Not as a request. CrawlCheck reads the rules your robots.txt sets for Cloudflare-AI-Search-External and reports them. Rules for it are reported and never scored, because refusing it is a policy choice.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [agentalog.com](/registry/capabilities/agentalog.com) | **Datasets:** [Idaho](/datasets/state/idaho) · [Technologies: Divi](/datasets/technologies/divi) · [Datasets](/datasets) | **Blog:** [areaServed vs service area pages: 20 towns declared, 1 page](/blog/your-schema-lists-service-areas-your-sitemap-has-pages-for-a-fraction-of-them) | **AI crawlers:** [Pinterestbot](/ai-crawlers/pinterestbot) | **Glossary:** [AI classifiers: Shortlisting](/glossary/ai-classifiers/shortlisting) · [Parsing](/glossary/parsing)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
