noindex
An instruction to keep a page out of search results, sent as a header or a meta tag.
What noindex does not show
A crawler obeys either, so setting it in one place and not the other still removes the page.
How CrawlCheck measures noindex
One finding code measures it. Its share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
ROBOTS_NOINDEX_LINE | 0.1% | dataset |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where noindex comes up on this site
- XML sitemap audit: check the file and every URL in itThe response may be an HTML login wall, a cached error page, a challenge interstitial, malformed XML, or a clean list of URLs that redirect, canonicalize elsewhere, carry noindex , or no longer exist.
- Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexingSites fail this audit by putting a rule in the wrong layer, and the most common failure is a noindex that no crawler can ever read. .t{font:600 13px system-ui,sans-serif;fill:var(--ink)}.s{font:12px…
- Staging site leak: how to find a staging URL in your live pages and close itAny crawler that follows the reference reaches a host that usually answers 200 with a full copy of the site, and unless that host sends noindex or requires authentication, it is a duplicate of every page with a different hostname.
- Content-Signal in robots.txt: search, ai-input, ai-trainPage-level directives such as noindex , nosnippet or max-snippet , in a meta robots tag or an X-Robots-Tag header, decide what a search engine may do with a fetched page, and Google applies nosnippet to its AI features as well as classic snippets.
- IndexNow: which engines use it and how to check it workedWhat IndexNow does not do It does not make a page indexable that is blocked by robots.txt, marked noindex , or canonicalised elsewhere; the engine will fetch it and reach the same conclusion it would have reached on its own.
Terms this definition uses
Definitions that name noindex
Related terms in Indexing and discovery
Whether a page can be found and kept, separately from whether it can be fetched.
nosnippet · max-snippet · max-image-preview · unavailable_after · noimageindex · index coverage · canonical · X-Robots-Tag · orphan page · E-E-A-T · E-E-A-T proxies · topical authority · Soft 404 · case consistency · accessibility structure · landmark region · accessible name · skip link · main landmark · alt attribute · form control name · duplicate id · IndexNow · faceted navigation · thin content · duplicate content · near-duplicate · boilerplate · MinHash · shingle · dead anchor · placeholder content · image placeholder · alt text · EXIF
Questions about noindex
What is noindex?
An instruction to keep a page out of search results, sent as a header or a meta tag.
What does noindex not show or guarantee?
A crawler obeys either, so setting it in one place and not the other still removes the page.
How does CrawlCheck measure noindex?
Through the finding code ROBOTS_NOINDEX_LINE, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does noindex belong to?
Indexing and discovery: Whether a page can be found and kept, separately from whether it can be fetched.
← canonical · X-Robots-Tag →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.