CrawlCheck

Guides · 2026-10-02 · By · 3 views

Content-Signal in robots.txt: search, ai-input, ai-train

Few top sites use Content-Signal yet: 4.3% of the top 10,000 in our census carry the line. What each value means, why it blocks nothing, and the one setting that costs you citations.

Content-Signal is a robots.txt line, introduced in Cloudflare's Content Signals Policy in September 2025, that states preferences for three uses of content: search (indexing and showing links and short excerpts), ai-input (using content in AI answers at answer time) and ai-train (training or fine-tuning models), each yes or no. An absent value neither grants nor refuses. It is not an access rule; User-agent and Disallow lines still decide what is fetched. Refusing ai-train costs no reach or citations; refusing ai-input or search asks answer engines not to use or show the page.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Content-Signal is still rare. In our census of the top 10,000 sites, 4.3% carry a Content-Signal line in robots.txt, and 133 sites set ai-train=no. Most of the web has not stated a preference either way, which means most sites are relying on crawl rules alone to say what they allow.

Robots.txt answers one question: may this crawler fetch this path. It cannot say "fetch this for search, but do not train on it". Content-Signal adds that second dimension as a single line stating which uses are permitted once the content has been fetched.

Content-Signal in the top 10,000Census result
Sites carrying a Content-Signal line4.3%
Sites setting ai-train=no133
VALUECOVERSIF SET TO NOsearchindex, links, short excerptsdropped from classic resultsai-inputgrounding and live AI answersnot quoted in AI answersai-traintraining or fine-tuningno effect on reach or citationA signal is a stated preference. The User-agent and Disallow rules decide what can be fetched.
Only refusing search or ai-input costs visibility. Refusing ai-train does not.

What is the syntax? #

The line sits in robots.txt, typically inside the User-agent: * group, as comma-separated pairs:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Under Cloudflare's policy, published on 24 September 2025, search covers building an index and returning links and short excerpts, and excludes AI-generated search summaries. ai-input covers putting content into a model at answer time: retrieval, grounding and real-time use in generative answers. ai-train covers training or fine-tuning. Each takes yes or no; a value left out neither grants nor restricts that use.

Does Content-Signal block anything? #

No. A crawler that ignores the line can fetch every page the User-agent rules allow. If you need a use prevented rather than discouraged, the crawl rules must do it: disallow the training crawler by name. The signal and the rules should also agree. A file that says ai-train=no while giving GPTBot an unconditional Allow: / states one policy and enforces another. Our own robots.txt carries a comment to that effect: every value reads yes because everything is fetchable, and flipping ai-train to no would mean disallowing the training crawlers as well.

Which setting costs you visibility? #

Signal setEffect on being foundEffect on being quoted in AI answersShown as an AI opt-out in a scan
no line at allnonenoneno
ai-train=no onlynonenoneno
ai-input=nononeremoved, for engines that honour ityes
search=noremoved from classic results, for engines that honour itusually removed tooyes
signal says no, rules allow everythingdepends on the crawlerdepends on the crawlerthe rules are what is enforced

The line that matters runs between training and answering. Refusing training is a coherent position that costs no measurable reach: an engine can still reach, read and quote the site. Refusing ai-input asks answer engines not to use the page when composing an answer, which is how a page gets cited. An earlier version of our report flagged ai-train=no as a problem too; that was wrong, because it penalised a deliberate and harmless choice, and the report no longer does it. Refusing ai-input or search shows up on 1.8% of scanned sites.

Who honours the signal? #

It is a published policy with a reservation-of-rights statement, not a protocol with mandatory compliance. Whether an operator honours it is that operator's decision, and most have not said publicly. Treat it as a clear statement of intent and use crawl rules for anything you need enforced. Intermediaries can also set a signal for you: Cloudflare's Markdown for Agents conversion adds a content-signal header reading yes for all three uses when the origin has not set one, as covered in serving Markdown to agents.

Which values should I set? #

If you want to be cited by answer engines, leave search and ai-input at yes. Set ai-train according to your view on model training; it does not affect citation. If you refuse training, back it with a Disallow: / group for the crawlers whose operators say they train, such as GPTBot and CCBot, plus Google's Google-Extended token, and make sure those groups do not drop your other rules, as explained in how named robots.txt groups work.

How do signals, robots rules and meta tags fit together? #

They answer different questions. User-agent and Disallow lines decide whether a path may be fetched. Page-level directives such as noindex, nosnippet or max-snippet, in a meta robots tag or an X-Robots-Tag header, decide what a search engine may do with a fetched page, and Google applies nosnippet to its AI features as well as classic snippets. Content signals state site-wide preferences for three categories of use. When they conflict, the most restrictive mechanism a given crawler honours applies. More in meta robots and X-Robots-Tag.

What mistakes are common? #

  • Putting the line before the first User-agent line, where a parser that reads only within groups may ignore it.
  • Spelling a value ai_train or aitrain, which matches nothing.
  • Setting ai-input=no while meaning to refuse only training, which removes the site from AI answers it wanted to appear in.
  • Setting a signal in robots.txt while a CDN or plugin sends a different content-signal header on every page.

How do I check what my site says? #

Fetch /robots.txt as a crawler would, not from a browser tab that may show a cached copy. Read the Content-Signal line and the groups around it, then check the response headers of a normal page for a content-signal header that might say something different. A free scan reports the signal next to each AI crawler's effective rules.

Share of scanned sites with each finding in this articleAI training opt-out set1.8%Share of scanned sites with eachfinding in this articleAI training opt-out set1.8%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What is Content-Signal in robots.txt?

A line stating which uses of a site's content are permitted after it is fetched: search, ai-input and ai-train, each yes or no. It comes from Cloudflare's Content Signals Policy, published in September 2025.

What is the difference between ai-input and ai-train?

ai-input covers using content at answer time, such as retrieval and grounding for AI answers. ai-train covers training or fine-tuning a model.

Does Content-Signal block crawlers?

No. It is a stated preference. Whether a page can be fetched is still decided by the User-agent, Allow and Disallow rules.

Will ai-train=no stop my site being cited by ChatGPT or Perplexity?

No. Refusing training does not affect reaching, reading or quoting a page. Refusing ai-input or search is what can remove a site from AI answers.

What does it mean if a signal is missing?

Under the policy, an absent value neither grants nor restricts that use.

How do I enforce a refusal to train?

Disallow the training crawlers by name in robots.txt, for example GPTBot, CCBot and Google-Extended, and repeat your other rules inside those groups.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works