Guides · 2026-09-16 · By VSNARY | Emmanuel Orta · 0 views

Meta robots, X-Robots-Tag and robots.txt: the three permission layers and the rule placed in the wrong one

Three mechanisms share a name and answer three different questions at three different moments. Most robots failures are a directive placed in a layer that cannot read it, starting with the noindex behind a Disallow.

Three different mechanisms are all called "robots", and they answer three different questions at three different moments. robots.txt is read before a request and decides whether a URL may be fetched at all. The meta robots tag and the X-Robots-Tag header ride on the response and decide what may be done with a page that was fetched: indexed, followed, quoted in a snippet, shown as an image. The newer AI signals, Content-Signal in robots.txt and the ai-train and ai-input vocabulary, decide what a fetched page may be used for. Sites fail this audit by putting a rule in the wrong layer, and the most common failure is a noindex that no crawler can ever read.

The short answer #

Use robots.txt to keep crawlers off URLs that should never be requested: admin paths, endless parameter spaces, private files. Use meta robots or X-Robots-Tag to keep fetched pages out of indexes and snippets. Never combine the two on the same URL: a Disallow stops the fetch, so the noindex on that page is never seen, and the URL can stay indexed from links alone. Then check that every named user-agent group in robots.txt carries the rules you think it does, because a named group replaces the default group rather than adding to it.

Layer 1: robots.txt is about fetching, not indexing #

A Disallow line tells a compliant crawler not to request the URL. It says nothing about whether the URL may appear in results; search engines index disallowed URLs from anchor text and external links, with no title and no snippet, for years. That is the source of the oldest robots mistake on the web: a page the owner wanted gone is disallowed, the crawler stops fetching it, the noindex the owner added later is behind the Disallow and is never read, and the URL remains. The file also has status rules of its own. Served as HTML it is a file with no rules; served as a 5xx it is treated by Google as a temporary full disallow; served as a 4xx it is treated as absent, which allows everything. The status-code guide covers those.

The failure that is specific to this layer, and the one most audits miss, is the group rule. A crawler reads the most specific User-agent group that matches it and ignores every other group, including *. So a file that says User-agent: * / Disallow: /admin/ and then adds User-agent: GPTBot / Allow: / to welcome an AI crawler has, for GPTBot, deleted the admin rule: the named group is the only one it reads, and that group allows everything. The scanner reports each named group that silently drops the default rules as ROBOTS_RULES_SHADOWED. It is the most common robots defect in the dataset, and it is manufactured by the very edits people make to be AI-friendly.

Layer 2: meta robots and X-Robots-Tag are about the fetched page #

Both carry the same directives, noindex, nofollow, nosnippet, noimageindex, max-snippet, unavailable_after, and a crawler applies whichever it finds. The tag lives in the HTML head and is only read on HTML pages. The header lives on the response and works on anything: PDFs, images, feeds, API endpoints, and also on HTML when the template cannot be edited. A directive in either place is honoured only if the crawler fetched the page, which is why it must not sit behind a Disallow.

The check on this layer is per page and per template, not per site. On a depth crawl the scanner records the noindex state of every page it reaches through the site's own navigation, alongside its canonical and its status. A page that is linked from the nav, answers 200 and says noindex is not necessarily wrong, but a whole template saying it is a section removed from the index by one line in a theme, and it is invisible from the homepage. The other thing to read is the crawler-specific form: <meta name="googlebot" ...> or an X-Robots-Tag with an agent prefix applies to that agent only, and a page can be indexable for one engine and not another without anything on the page looking odd.

Layer 3: the AI-use signals #

The third vocabulary is newer and lives in robots.txt again, but it answers a different question. A Content-Signal line, and the ai-train / ai-input / search values it carries, tell an agent what a page it is allowed to fetch may be used for: search results, AI answers, model training. The scanner records a Content-Signal that refuses AI answers or search as AI_OPTOUT_SET, and separately reads whether the signal actually reaches the agents it is meant for. Because it is a robots.txt directive, it is subject to the same group rule as everything else in the file: a Content-Signal: ai-train=no under * never reaches an agent that has its own named group, and if that group allows the agent in, the site has said no to training in a place the training crawler does not read. That contradiction is recorded as INSTRUCTION_CONFLICT, with the signal, the group and the agents it fails to reach.

The collisions, in the order they happen #

Disallow plus noindex. The noindex is unreachable. Remove the Disallow, let the crawler fetch the page once and read the noindex, and only then, if you still want the URL never requested, restore the Disallow. Disallow plus canonical. Same shape: a canonical on a page the crawler may not fetch is a hint it never receives. Named group plus default rules. Any rule you want to apply to a named agent must be repeated inside its group; the default group does not inherit downward. Header plus tag. When both are present and disagree, crawlers apply the more restrictive directive, so an X-Robots-Tag: noindex set at the edge for one path pattern will override an index tag on every page it matches, silently. Signal under * plus named agent. Repeat the Content-Signal inside every named group, or drop the named groups.

Test it yourself #

Fetch robots.txt and read the content type before the rules. List every User-agent group; for each named agent, read only its own group and ask whether the rules you care about are inside it. Fetch a page you believe is noindexed and read both the meta tag and the X-Robots-Tag header on the final response; if the URL is disallowed, note that no compliant crawler will ever see what you just read. Fetch a PDF or image you want kept out and check that the header, not a tag, carries the directive. Then fetch a page you want indexed with a crawler user-agent and confirm neither the tag nor the header says otherwise for that agent.

The free scan resolves every named agent in robots.txt into what it is actually permitted to do, reads the noindex state per page on a depth crawl, and reports the group and signal conflicts above. Do AI crawlers respect robots.txt covers which agents honour which layer, and the instructions on your domain you did not write is the same audit run on files a site did not know it was serving.

Fix order #

robots.txt status and content type first, because a file that is not read as a file makes every other layer moot. Then the shadowed groups, because they change what every named crawler is allowed to fetch. Then any Disallow that sits in front of a noindex or a canonical, because those directives are dead until the fetch is allowed. Then the per-template noindex on the crawl. The use signals last: they are the least widely honoured and the most likely to be rewritten as the vocabulary settles, and they are only worth placing once the groups they have to reach are correct.

Questions this post answers

Does Disallow in robots.txt remove a page from search results?

No. Disallow stops the fetch. Search engines can still index the URL from links, without a title or snippet. Removing a page needs a noindex the crawler can read, which means the URL must be fetchable when the noindex is added.

What is the difference between meta robots and X-Robots-Tag?

They carry the same directives. The meta tag lives in the HTML head and only works on HTML pages. The X-Robots-Tag header rides on the response and works on any file type, including PDFs and images, and can be set at the edge for a path pattern.

Why does adding a User-agent group for an AI crawler break my other rules?

A crawler reads the most specific group that matches it and ignores every other group, including the default. A named group that only says Allow has, for that agent, replaced the whole default rule set. Repeat the rules inside the named group.

Where do AI training opt-outs go?

In robots.txt, as a Content-Signal line carrying ai-train, ai-input or search values. Because it is a robots.txt directive, it follows the group rule: a signal under the default group does not reach an agent that has its own named group.

What happens when the meta tag and the header disagree?

Crawlers apply the more restrictive directive. An X-Robots-Tag noindex set at the edge for a path pattern overrides an index tag on every page it matches, which is why a noindex should be checked on the final response headers, not only in the HTML.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works