Guides · 2026-09-16 · By VSNARY | Emmanuel Orta
Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexing
Three mechanisms share a name and answer three different questions at three different moments. Most robots failures are a directive placed in a layer that cannot read it, starting with the noindex behind a Disallow.
robots.txt controls fetching: it is read before a request and decides whether a URL may be crawled at all. The meta robots tag and the X-Robots-Tag header control what happens to a page that was fetched: whether it is indexed, followed or shown in a snippet. The tag sits in the HTML, and the header also works for PDFs and images. The most common mistake is a noindex on a page that robots.txt disallows: the crawler never fetches the page, so it never sees the noindex, and the URL can stay in the index.
Three different mechanisms are all called "robots", and they answer three different questions at three different moments. robots.txt is read before a request and decides whether a URL may be fetched at all. The meta robots tag and the X-Robots-Tag header ride on the response and decide what may be done with a page that was fetched: indexed, followed, quoted in a snippet, shown as an image. The newer AI signals, Content-Signal in robots.txt and the ai-train and ai-input vocabulary, decide what a fetched page may be used for. Sites fail this audit by putting a rule in the wrong layer, and the most common failure is a noindex that no crawler can ever read.
The short answer #
Use robots.txt to keep crawlers off URLs that should never be requested: admin paths, endless parameter spaces, private files. Use meta robots or X-Robots-Tag to keep fetched pages out of indexes and snippets. Never combine the two on the same URL: a Disallow stops the fetch, so the noindex on that page is never seen, and the URL can stay indexed from links alone. Then check that every named user-agent group in robots.txt carries the rules you think it does, because a named group replaces the default group rather than adding to it.
Layer 1: robots.txt is about fetching, not indexing #
A Disallow line tells a compliant crawler not to request the URL. It says nothing about whether the URL may appear in results; search engines index disallowed URLs from anchor text and external links, with no title and no snippet, for years. That is the source of the oldest robots mistake on the web: a page the owner wanted gone is disallowed, the crawler stops fetching it, the noindex the owner added later is behind the Disallow and is never read, and the URL remains. The file also has status rules of its own. Served as HTML it is a file with no rules; served as a 5xx it is treated by Google as a temporary full disallow; served as a 4xx it is treated as absent, which allows everything. The status-code guide covers those.
The failure that is specific to this layer, and the one most audits miss, is the group rule. A crawler reads the most specific User-agent group that matches it and ignores every other group, including *. So a file that says User-agent: * / Disallow: /admin/ and then adds User-agent: GPTBot / Allow: / to welcome an AI crawler has, for GPTBot, deleted the admin rule: the named group is the only one it reads, and that group allows everything. Check every named group: each one that silently drops the default rules is a rule you wrote that never applies. It is one of the most common robots defects in the dataset, and it is manufactured by the very edits people make to be AI-friendly.
Layer 2: meta robots and X-Robots-Tag are about the fetched page #
Both carry the same directives, noindex, nofollow, nosnippet, noimageindex, max-snippet, unavailable_after, and a crawler applies whichever it finds. The tag lives in the HTML head and is only read on HTML pages. The header lives on the response and works on anything: PDFs, images, feeds, API endpoints, and also on HTML when the template cannot be edited. A directive in either place is honoured only if the crawler fetched the page, which is why it must not sit behind a Disallow.
The check on this layer is per page and per template, not per site. Crawl the site through its own navigation and record every page's noindex state, canonical and status. A page that is linked from the nav, answers 200 and says noindex is not necessarily wrong, but a whole template saying it is a section removed from the index by one line in a theme, and it is invisible from the homepage. The other thing to read is the crawler-specific form: <meta name="googlebot" ...> or an X-Robots-Tag with an agent prefix applies to that agent only, and a page can be indexable for one engine and not another without anything on the page looking odd.
Layer 3: the AI-use signals #
The third vocabulary is newer and lives in robots.txt again, but it answers a different question. A Content-Signal line, and the ai-train / ai-input / search values it carries, tell an agent what a page it is allowed to fetch may be used for: search results, AI answers, model training. If your Content-Signal refuses AI answers or search, check that it was a decision, and that the line actually reaches the agents it is meant for. Because it is a robots.txt directive, it is subject to the same group rule as everything else in the file: a Content-Signal: ai-train=no under * never reaches an agent that has its own named group, and if that group allows the agent in, the site has said no to training in a place the training crawler does not read. A scan reports that contradiction with the signal, the group and the agents it fails to reach.
| Failure | Layer | Share of scans |
|---|---|---|
A named User-agent group silently drops the rules under * | 1 · fetch | 13% |
| robots.txt disallows everything for every agent | 1 · fetch | 1.1% |
| robots.txt answers with an HTML document | 1 · fetch | 0.1% |
| A Content-Signal refuses AI answers or search | 3 · use | 2.9% |
A use signal under * never reaches the agent it targets | 1 and 3 | 0.4% |
The collisions, in the order they happen #
Disallow plus noindex. The noindex is unreachable. Remove the Disallow, let the crawler fetch the page once and read the noindex, and only then, if you still want the URL never requested, restore the Disallow. Disallow plus canonical. Same shape: a canonical on a page the crawler may not fetch is a hint it never receives. Named group plus default rules. Any rule you want to apply to a named agent must be repeated inside its group; the default group does not inherit downward. Header plus tag. When both are present and disagree, crawlers apply the more restrictive directive, so an X-Robots-Tag: noindex set at the edge for one path pattern will override an index tag on every page it matches, silently. Signal under * plus named agent. Repeat the Content-Signal inside every named group, or drop the named groups.
Test it yourself #
Fetch robots.txt and read the content type before the rules. List every User-agent group; for each named agent, read only its own group and ask whether the rules you care about are inside it. Fetch a page you believe is noindexed and read both the meta tag and the X-Robots-Tag header on the final response; if the URL is disallowed, note that no compliant crawler will ever see what you just read. Fetch a PDF or image you want kept out and check that the header, not a tag, carries the directive. Then fetch a page you want indexed with a crawler user-agent and confirm neither the tag nor the header says otherwise for that agent.
The free scan resolves every named agent in robots.txt into what it is actually permitted to do, reads the noindex state per page on a depth crawl, and reports the group and signal conflicts above. Do AI crawlers respect robots.txt covers which agents honour which layer, and the instructions on your domain you did not write is the same audit run on files a site did not know it was serving.
Fix order #
robots.txt status and content type first, because a file that is not read as a file makes every other layer moot. Then the shadowed groups, because they change what every named crawler is allowed to fetch. Then any Disallow that sits in front of a noindex or a canonical, because those directives are dead until the fetch is allowed. Then the per-template noindex on the crawl. The use signals last: they are the least widely honoured and the most likely to be rewritten as the vocabulary settles, and they are only worth placing once the groups they have to reach are correct.
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Questions this post answers
How do I keep a PDF out of Google search results?
Send an X-Robots-Tag: noindex header on the PDF's response, because a meta robots tag can only sit inside an HTML page. Set the header on the server or at the edge, for one file or a pattern such as every PDF in a folder. Do not also Disallow the file in robots.txt, or crawlers never fetch it and never see the header.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.