Agent view · Guides · 2026-10-02
Content-Signal in robots.txt: search, ai-input, ai-train
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/content-signals-in-robots-txt-search-ai-input-ai-train: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/content-signals-in-robots-txt-search-ai-input-ai-train>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 15,724 |
| words | 2,200 |
| sha-256 | fb808322f3179af5ead732a2c4e734fd0fe7251dbf4e1db4199f611d51b56e3e |
| repr-digest | sha-256=:+4CDIvMXmvXq1zKixOc0/Q/nJR2/Th20GZ9hHVG1bj4=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311496;expires=1795315096;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="oHQA8nd3SyhdpCpcTnyc7g";tag="crawlcheck-page" |
| signature | sig1=:N5f5DfDujSe5zFdiwvq6mfWCIT3No8ZFQw3D0FX+5GCKQWw9OTkjgg72LoYdKQy8suRB4juWrJOi7tMngDoeAA==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-02 · By [VSNARY | Emmanuel Orta](/author) · 3 views # Content-Signal in robots.txt: search, ai-input, ai-train Few top sites use Content-Signal yet: 4.3% of the top 10,000 in our census carry the line. What each value means, why it blocks nothing, and the one setting that costs you citations. Content-Signal is a robots.txt line, introduced in Cloudflare's Content Signals Policy in September 2025, that states preferences for three uses of content: search (indexing and showing links and short excerpts), ai-input (using content in AI answers at answer time) and ai-train (training or fine-tuning models), each yes or no. An absent value neither grants nor refuses. It is not an access rule; User-agent and Disallow lines still decide what is fetched. Refusing ai-train costs no reach or citations; refusing ai-input or search asks answer engines not to use or show the page. Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - What is the syntax? - Does Content-Signal block anything? - Which setting costs you visibility? - Who honours the signal? - Which values should I set? - How do signals, robots rules and meta tags fit together? - What mistakes are common? - How do I check what my site says? [Content-Signal](/glossary/content-signal) is still rare. In our census of the top 10,000 sites, **4.3%** carry a Content-Signal line in [robots.txt](/glossary/robots-txt), and **133** sites set `ai-train=no`. Most of the web has not stated a preference either way, which means most sites are relying on crawl rules alone to say what they allow. Robots.txt answers one question: may this crawler fetch this path. It cannot say "fetch this for search, but do not train on it". Content-Signal adds that second dimension as a single line stating which uses are permitted once the content has been fetched. | Content-Signal in the top 10,000 | Census result | | --- | --- | | Sites carrying a Content-Signal line | 4.3% | | Sites setting ai-train=no | 133 | Only refusing search or ai-input costs visibility. Refusing ai-train does not. ## What is the syntax? # The line sits in robots.txt, typically inside the `User-agent: *` group, as comma-separated pairs: ``` User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / ``` Under Cloudflare's policy, published on 24 September 2025, `search` covers building an index and returning links and short excerpts, and excludes AI-generated search summaries. `ai-input` covers putting content into a model at answer time: [retrieval](/glossary/retrieval), [grounding](/glossary/grounding) and real-time use in generative answers. `ai-train` covers training or fine-tuning. Each takes `yes` or `no`; a value left out neither grants nor restricts that use. ## Does Content-Signal block anything? # No. A crawler that ignores the line can fetch every page the `User-agent` rules allow. If you need a use prevented rather than discouraged, the crawl rules must do it: disallow the training crawler by name. The signal and the rules should also agree. A file that says `ai-train=no` while giving GPTBot an unconditional `Allow: /` states one policy and enforces another. Our own robots.txt carries a comment to that effect: every value reads yes because everything is fetchable, and flipping `ai-train` to no would mean disallowing the training crawlers as well. ## Which setting costs you visibility? # | Signal set | Effect on being found | Effect on being quoted in AI answers | Shown as an [AI opt-out](/glossary/ai-opt-out) in a scan | | --- | --- | --- | --- | | no line at all | none | none | no | | `ai-train=no` only | none | none | no | | `ai-input=no` | none | removed, for engines that honour it | yes | | `search=no` | removed from classic results, for engines that honour it | usually removed too | yes | | signal says no, rules allow everything | depends on the crawler | depends on the crawler | the rules are what is enforced | The line that matters runs between training and answering. Refusing training is a coherent position that costs no measurable reach: an engine can still reach, read and quote the site. Refusing `ai-input` asks answer engines not to use the page when composing an answer, which is how a page gets cited. An earlier version of our report flagged `ai-train=no` as a problem too; that was wrong, because it penalised a deliberate and harmless choice, and the report no longer does it. Refusing ai-input or search shows up on **1.8%** of scanned sites. ## Who honours the signal? # It is a published policy with a reservation-of-rights statement, not a protocol with mandatory compliance. Whether an operator honours it is that operator's decision, and most have not said publicly. Treat it as a clear statement of intent and use crawl rules for anything you need enforced. Intermediaries can also set a signal for you: Cloudflare's Markdown for Agents conversion adds a `content-signal` header reading yes for all three uses when the origin has not set one, as covered in [serving Markdown to agents](/guides/markdown-for-agents-serving-text-markdown-by-content-negotiation). ## Which values should I set? # If you want to be cited by answer engines, leave `search` and `ai-input` at yes. Set `ai-train` according to your view on model training; it does not affect [citation](/glossary/citation). If you refuse training, back it with a `Disallow: /` group for the crawlers whose operators say they train, such as GPTBot and [CCBot](/glossary/ccbot), plus Google's `Google-Extended` [token](/glossary/token), and make sure those groups do not drop your other rules, as explained in [how named robots.txt groups work](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules). ## How do signals, robots rules and meta tags fit together? # They answer different questions. `User-agent` and `Disallow` lines decide whether a path may be fetched. Page-level directives such as `noindex`, `nosnippet` or `max-snippet`, in a meta robots tag or an `X-Robots-Tag` header, decide what a search engine may do with a fetched page, and Google applies `nosnippet` to its AI features as well as classic snippets. Content signals state site-wide preferences for three categories of use. When they conflict, the most restrictive mechanism a given crawler honours applies. More in [meta robots and X-Robots-Tag](/guides/meta-robots-and-x-robots-tag-the-three-layers-of-crawler-permission). ## What mistakes are common? # - Putting the line before the first `User-agent` line, where a parser that reads only within groups may ignore it. - Spelling a value `ai_train` or `aitrain`, which matches nothing. - Setting `ai-input=no` while meaning to refuse only training, which removes the site from AI answers it wanted to appear in. - Setting a signal in robots.txt while a [CDN](/glossary/cdn) or plugin sends a different `content-signal` header on every page. ## How do I check what my site says? # Fetch `/robots.txt` as a crawler would, not from a browser tab that may show a cached copy. Read the `Content-Signal` line and the groups around it, then check the response headers of a normal page for a `content-signal` header that might say something different. A [free scan](/) reports the signal next to each AI crawler's effective rules. Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcontent-signals-in-robots-txt-search-ai-input-ai-train%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcontent-signals-in-robots-txt-search-ai-input-ai-train%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Content-Signal%20in%20robots.txt%3A%20search%2C%20ai-input%2C%20ai-train)[Email](mailto:?subject=Content-Signal%20in%20robots.txt%3A%20search%2C%20ai-input%2C%20ai-train&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcontent-signals-in-robots-txt-search-ai-input-ai-train)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### What is Content-Signal in robots.txt? A line stating which uses of a site's content are permitted after it is fetched: search, ai-input and ai-train, each yes or no. It comes from Cloudflare's Content Signals Policy, published in September 2025. ### What is the difference between ai-input and ai-train? ai-input covers using content at answer time, such as retrieval and grounding for AI answers. ai-train covers training or fine-tuning a model. ### Does Content-Signal block crawlers? No. It is a stated preference. Whether a page can be fetched is still decided by the User-agent, Allow and Disallow rules. ### Will ai-train=no stop my site being cited by ChatGPT or Perplexity? No. Refusing training does not affect reaching, reading or quoting a page. Refusing ai-input or search is what can remove a site from AI answers. ### What does it mean if a signal is missing? Under the policy, an absent value neither grants nor restricts that use. ### How do I enforce a refusal to train? Disallow the training crawlers by name in robots.txt, for example GPTBot, CCBot and Google-Extended, and repeat your other rules inside those groups. ## Related findings [2026-11-03 One Cloudflare setting is behind most of the new AI training blocks on top sites Census of the Tranco top 10,000, October to November 2026: which AI crawlers sites newly blocked, how much of it came from Cloudfl](/blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026)[2026-10-26 Google-Extended and Applebot-Extended: block AI training Google-Extended and Applebot-Extended are tokens, not crawlers: they opt a site out of AI training and keep it in search. The line](/guides/google-extended-and-applebot-extended-opt-out-of-ai-training-keep-search)[2026-10-26 Which AI crawlers the top 10,000 sites block most, October 2026 Top 10,000 sites, October 2026: robots.txt block rates for 12 AI crawlers, how many refuse them at the door, and why training is b](/blog/which-ai-crawlers-top-sites-block-most-october-2026) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [acxiom.com](/registry/acxiom.com) · [congress.gov](/registry/congress.gov) · [flippio.pl](/registry/flippio.pl) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) | **Glossary:** [Semantic chunking](/glossary/semantic-chunking) · [410 response](/glossary/410-response) · [Sponsored link](/glossary/sponsored-link) · [published crawler ranges](/glossary/published-crawler-ranges) · [Content-Digest](/glossary/content-digest) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/content-signals-in-robots-txt-search-ai-input-ai-train.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.