X-Robots-Tag
An HTTP header carrying indexing directives.
What X-Robots-Tag means
Invisible in the page source, which is why a page can look perfectly indexable in a browser while being excluded.
How CrawlCheck measures X-Robots-Tag
One finding code measures it. Its share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
INSTRUCTION_CONFLICT | 0.5% | Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexing |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where X-Robots-Tag comes up on this site
- Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexingThe meta robots tag and the X-Robots-Tag header ride on the response and decide what may be done with a page that was fetched: indexed, followed, quoted in a snippet, shown as an image.
- XML sitemap audit: check the file and every URL in itFlag any non- 200 final status, any redirect from a listed URL, suspiciously tiny or identical bodies, X-Robots-Tag: noindex or a meta noindex , a robots.txt disallow for the crawlers you want, and any canonical pointing elsewhere.
- Staging site leak: how to find a staging URL in your live pages and close itIf a host is found, fetch it directly and read three things: the status, whether an X-Robots-Tag: noindex or a meta robots tag is present, and whether it demands authentication.
- Content-Signal in robots.txt: search, ai-input, ai-trainPage-level directives such as noindex , nosnippet or max-snippet , in a meta robots tag or an X-Robots-Tag header, decide what a search engine may do with a fetched page, and Google applies nosnippet to its AI features as well as classic snippets.
Definitions that name X-Robots-Tag
nosnippet · max-snippet · noimageindex
Related terms in Indexing and discovery
Whether a page can be found and kept, separately from whether it can be fetched.
nosnippet · max-snippet · max-image-preview · unavailable_after · noimageindex · index coverage · canonical · noindex · orphan page · E-E-A-T · E-E-A-T proxies · topical authority · Soft 404 · case consistency · accessibility structure · landmark region · accessible name · skip link · main landmark · alt attribute · form control name · duplicate id · IndexNow · faceted navigation · thin content · duplicate content · near-duplicate · boilerplate · MinHash · shingle · dead anchor · placeholder content · image placeholder · alt text · EXIF
Questions about X-Robots-Tag
What is X-Robots-Tag?
An HTTP header carrying indexing directives.
How does CrawlCheck measure X-Robots-Tag?
Through the finding code INSTRUCTION_CONFLICT, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does X-Robots-Tag belong to?
Indexing and discovery: Whether a page can be found and kept, separately from whether it can be fetched.
← noindex · orphan page →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.