Guides · 2026-10-02 · By VSNARY | Emmanuel Orta · 3 views
Content-Signal in robots.txt: search, ai-input, ai-train
Few top sites use Content-Signal yet: 4.3% of the top 10,000 in our census carry the line. What each value means, why it blocks nothing, and the one setting that costs you citations.
Content-Signal is a robots.txt line, introduced in Cloudflare's Content Signals Policy in September 2025, that states preferences for three uses of content: search (indexing and showing links and short excerpts), ai-input (using content in AI answers at answer time) and ai-train (training or fine-tuning models), each yes or no. An absent value neither grants nor refuses. It is not an access rule; User-agent and Disallow lines still decide what is fetched. Refusing ai-train costs no reach or citations; refusing ai-input or search asks answer engines not to use or show the page.
Content-Signal is still rare. In our census of the top 10,000 sites, 4.3% carry a Content-Signal line in robots.txt, and 133 sites set ai-train=no. Most of the web has not stated a preference either way, which means most sites are relying on crawl rules alone to say what they allow.
Robots.txt answers one question: may this crawler fetch this path. It cannot say "fetch this for search, but do not train on it". Content-Signal adds that second dimension as a single line stating which uses are permitted once the content has been fetched.
| Content-Signal in the top 10,000 | Census result |
|---|---|
| Sites carrying a Content-Signal line | 4.3% |
| Sites setting ai-train=no | 133 |
What is the syntax? #
The line sits in robots.txt, typically inside the User-agent: * group, as comma-separated pairs:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /Under Cloudflare's policy, published on 24 September 2025, search covers building an index and returning links and short excerpts, and excludes AI-generated search summaries. ai-input covers putting content into a model at answer time: retrieval, grounding and real-time use in generative answers. ai-train covers training or fine-tuning. Each takes yes or no; a value left out neither grants nor restricts that use.
Does Content-Signal block anything? #
No. A crawler that ignores the line can fetch every page the User-agent rules allow. If you need a use prevented rather than discouraged, the crawl rules must do it: disallow the training crawler by name. The signal and the rules should also agree. A file that says ai-train=no while giving GPTBot an unconditional Allow: / states one policy and enforces another. Our own robots.txt carries a comment to that effect: every value reads yes because everything is fetchable, and flipping ai-train to no would mean disallowing the training crawlers as well.
Which setting costs you visibility? #
| Signal set | Effect on being found | Effect on being quoted in AI answers | Shown as an AI opt-out in a scan |
|---|---|---|---|
| no line at all | none | none | no |
ai-train=no only | none | none | no |
ai-input=no | none | removed, for engines that honour it | yes |
search=no | removed from classic results, for engines that honour it | usually removed too | yes |
| signal says no, rules allow everything | depends on the crawler | depends on the crawler | the rules are what is enforced |
The line that matters runs between training and answering. Refusing training is a coherent position that costs no measurable reach: an engine can still reach, read and quote the site. Refusing ai-input asks answer engines not to use the page when composing an answer, which is how a page gets cited. An earlier version of our report flagged ai-train=no as a problem too; that was wrong, because it penalised a deliberate and harmless choice, and the report no longer does it. Refusing ai-input or search shows up on 1.8% of scanned sites.
Who honours the signal? #
It is a published policy with a reservation-of-rights statement, not a protocol with mandatory compliance. Whether an operator honours it is that operator's decision, and most have not said publicly. Treat it as a clear statement of intent and use crawl rules for anything you need enforced. Intermediaries can also set a signal for you: Cloudflare's Markdown for Agents conversion adds a content-signal header reading yes for all three uses when the origin has not set one, as covered in serving Markdown to agents.
Which values should I set? #
If you want to be cited by answer engines, leave search and ai-input at yes. Set ai-train according to your view on model training; it does not affect citation. If you refuse training, back it with a Disallow: / group for the crawlers whose operators say they train, such as GPTBot and CCBot, plus Google's Google-Extended token, and make sure those groups do not drop your other rules, as explained in how named robots.txt groups work.
How do signals, robots rules and meta tags fit together? #
They answer different questions. User-agent and Disallow lines decide whether a path may be fetched. Page-level directives such as noindex, nosnippet or max-snippet, in a meta robots tag or an X-Robots-Tag header, decide what a search engine may do with a fetched page, and Google applies nosnippet to its AI features as well as classic snippets. Content signals state site-wide preferences for three categories of use. When they conflict, the most restrictive mechanism a given crawler honours applies. More in meta robots and X-Robots-Tag.
What mistakes are common? #
- Putting the line before the first
User-agentline, where a parser that reads only within groups may ignore it. - Spelling a value
ai_trainoraitrain, which matches nothing. - Setting
ai-input=nowhile meaning to refuse only training, which removes the site from AI answers it wanted to appear in. - Setting a signal in robots.txt while a CDN or plugin sends a different
content-signalheader on every page.
How do I check what my site says? #
Fetch /robots.txt as a crawler would, not from a browser tab that may show a cached copy. Read the Content-Signal line and the groups around it, then check the response headers of a normal page for a content-signal header that might say something different. A free scan reports the signal next to each AI crawler's effective rules.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is Content-Signal in robots.txt?
A line stating which uses of a site's content are permitted after it is fetched: search, ai-input and ai-train, each yes or no. It comes from Cloudflare's Content Signals Policy, published in September 2025.
What is the difference between ai-input and ai-train?
ai-input covers using content at answer time, such as retrieval and grounding for AI answers. ai-train covers training or fine-tuning a model.
Does Content-Signal block crawlers?
No. It is a stated preference. Whether a page can be fetched is still decided by the User-agent, Allow and Disallow rules.
Will ai-train=no stop my site being cited by ChatGPT or Perplexity?
No. Refusing training does not affect reaching, reading or quoting a page. Refusing ai-input or search is what can remove a site from AI answers.
What does it mean if a signal is missing?
Under the policy, an absent value neither grants nor restricts that use.
How do I enforce a refusal to train?
Disallow the training crawlers by name in robots.txt, for example GPTBot, CCBot and Google-Extended, and repeat your other rules inside those groups.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.