CrawlCheck

Guides · 2026-09-16 · By · 0 views

Canonical URL audit: one page, one address, and the three ways a site publishes it twice

The canonical tag is one line and most audits stop at it. The audit that matters counts how many addresses answer 200 for the same bytes, whether the sitemap agrees, and whether the hostname is a site at all.

Share of all scans carrying each finding named aboveHOST_DUPLICATE_2002.8%HOMEPAGE_REDIRECTS_OFF_HOST1.9%DOMAIN_IS_ALIAS0.9%STAGING_LEAK0.1%Share of all scans carrying eachfinding named aboveHOST_DUPLICATE_2002.8%HOMEPAGE_REDIRECTS_OFF_HOST1.9%DOMAIN_IS_ALIAS0.9%STAGING_LEAK0.1%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

A canonical URL is the one address a site asks crawlers to treat as the page. The tag that declares it is one line, and most audits check that the line exists. The audit that matters is different: how many addresses actually answer 200 for the same content, whether each of them names the same canonical, whether the site's own machine files agree, and whether the hostname a crawler was sent to is the site at all or an alias of another one. Each of those is measurable from the responses, and each fails in a way a tag check cannot see.

http://example.com/301 → https, same hosthttps://www.example.com/200, stays on wwwhttps://example.com/200, canonical = selfhttps://old-brand.com/every path 301 → other hostone pagefour ways to reach ithousekeeping: not a findingtwo hosts serve the page: HOST_DUPLICATE_200the address the site names for itselfan alias, not a site: DOMAIN_IS_ALIAS
A canonical audit is not about the tag. It is about how many addresses answer 200 for the same bytes, and whether every one of them agrees on which address is the page.

The short answer #

One page should have one address that answers 200, and every other way of reaching it should redirect there in one hop. The canonical tag should name that address, with the same scheme, host and path the server actually serves. The sitemap should list that address and no other. If any two of those disagree, a crawler has to pick, and it will not always pick the one you meant.

Why a tag check is not an audit #

The canonical link element is a hint, not an instruction. Search engines weigh it against redirects, internal links, sitemap entries and the addresses other sites link to, and when the signals disagree the tag loses. A page whose tag says https://example.com/services/ while the server answers 200 on https://www.example.com/services/, the sitemap lists the trailing-slash-free version and the nav links the www one has declared four canonicals, and only one of them is in the tag. The check that finds this is not "is there a tag" but "do the tag, the response, the sitemap and the links all name the same address".

Host duplicates: two addresses, one set of bytes #

The most common canonical failure is not on any page. It is the hostname. A site that answers 200 on both example.com and www.example.com, and stays on whichever one was requested, has published every page twice. The scanner requests the www form of the homepage and records whether the final URL is still on www and whether the byte count is within fifteen percent of the apex response. Both true means the two hosts are serving the same page rather than one redirecting to the other, recorded as HOST_DUPLICATE_200. The canonical tag may be correct on both copies; the duplicate exists anyway, because a crawler that arrives via a link to the wrong host is served a full page rather than a redirect, and has to decide for itself which host is real.

The same failure hides behind schemes and trailing slashes. http:// should 301 to https:// on the same host in one hop, never serve a page. A path should answer 200 with or without its trailing slash, never both. Each of these is one request to check and one server rule to fix, and each one left open is a second address competing with the first.

Aliases: the hostname that is not a site #

The opposite failure is a hostname that answers nothing of its own. An old brand domain, a defensive registration, a market-specific domain pointed at the main site: every path redirects to another host, including the machine files. The scanner reads the homepage redirect together with where robots.txt, the sitemap and the other machine files land; when all of them redirect to the same host, the domain is recorded as DOMAIN_IS_ALIAS, and the off-host homepage redirect that would otherwise be a defect is downgraded, because under an alias it is the intended behaviour. What matters for the audit is that the alias is complete: a domain that redirects its homepage but serves its own robots.txt is half an alias, and a crawler that reads the file gets a policy for a site that does not exist there.

Off-host redirects on a live site #

Between a duplicate and an alias sits the site whose homepage ends on another host while the rest of the site stays. That is recorded as HOMEPAGE_REDIRECTS_OFF_HOST, and it is the one to check by hand before believing it, because the two explanations look identical from the status line: a migration in progress, or a domain that was meant to be an alias and is only partly configured. The redirect chain's final host tells you where the entity lives now; the canonical tags on the remaining pages tell you whether the site knows.

The canonical on every page of the crawl #

On a depth crawl the scanner records, for every page it fetches, whether the canonical tag names the page's own final URL. A canonical that points elsewhere is not wrong on its face: paginated lists, tracking parameters and print views legitimately point at a parent. But a page that a crawler reached through the site's own navigation, that answers 200, and whose canonical names a different URL has told the crawler that it is not the page to index. When that happens across a template it is usually one line in the theme, and it silently removes a whole section from the index while every page continues to answer 200.

FailureHow it is measuredFindingShare of scans
Both hosts serve the pagewww request stays on www and returns bytes within 15% of the apexHOST_DUPLICATE_2002.8%
Homepage ends on another hostRedirect chain followed to its final hostHOMEPAGE_REDIRECTS_OFF_HOST1.9%
Every path, machine files included, redirects to one other hostHomepage plus robots.txt, sitemap and the other machine files all land on the same hostDOMAIN_IS_ALIAS0.9%
A staging or preview host referenced in the page sourceHostnames in the source that name a staging environmentSTAGING_LEAK0.1%

The sitemap has to agree #

A sitemap is the site's own list of canonical addresses, and it is the place where a canonical disagreement is easiest to catch, because it is a single file listing every URL the site wants indexed. Fetch every listed URL and compare its canonical tag with the sitemap entry. A listed URL whose canonical points somewhere else is a page the site has declared twice with two different answers. A listed URL that redirects has a sitemap that is out of date. The sitemap audit guide walks through that comparison layer by layer; this guide is the half of it that starts from the page instead of the file.

Parameters and the addresses you did not mean to publish #

Every tracking parameter, session id and sort option that reaches a 200 page is a canonical decision the site made without noticing. If the parameterised address serves the page with a self-referencing canonical, it is a duplicate. If it serves the page with a canonical pointing at the clean URL, it is a hint the crawler may or may not take. The safe shape is the one where the parameter never produces a distinct 200 page at all: strip it at the edge, or answer it with the canonical of the clean address and keep the clean address the only one that internal links use. The crawl-waste rows in a report count how many of a site's own links carry a query string, because that is where the parameterised duplicates are manufactured.

Test it yourself #

Request the four homepage forms, http and https on the apex and on www, and record the final URL and status of each. Three should be one-hop redirects to the fourth. Request a handful of inner pages on the wrong host and confirm they redirect rather than serve. Read the canonical tag on each page the redirect lands on and confirm it names the landed URL exactly, scheme and slash included. Open the sitemap and spot-check that its entries are the same addresses. Then request one page with a made-up query parameter and see whether it answers with the page, a redirect, or the page with a canonical to the clean URL; only the last two are safe.

The free scan runs the host and alias checks on every domain and the per-page canonical comparison on a depth crawl, and the status-code guide covers how to read the redirect chains those checks depend on.

Fix order #

Host first: one hostname answers, the other three forms redirect to it in one hop. Then any off-host homepage redirect, which is either a migration to finish or an alias to complete. Then the template canonical, checked on one page per template rather than one page per site. Then the sitemap, regenerated from the canonical addresses rather than edited by hand. Parameters last: they are the most numerous and the least damaging, and the edge rule that strips them is the same rule that fixes crawl waste.

Every figure above came out of this scanner.

Point it at your own domain and see the same measurements, free.

Scan a domain — free

The main product

Found this on your own site? We fix it for $749.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Questions this post answers

Is a canonical tag enough to fix duplicate content?

No. The tag is a hint that search engines weigh against redirects, sitemap entries and internal links. If the server answers 200 on two hosts, or the sitemap lists a different address, the tag can be overruled. The fix is to make one address the only one that answers 200 and to have every other signal name it.

What is a host duplicate?

A site that answers 200 on both the apex domain and the www form, staying on whichever was requested, has published every page at two addresses. The scanner records it when the www request stays on www and returns a byte count within fifteen percent of the apex.

What does it mean when a domain is an alias?

Every path on the domain, machine files included, redirects to the same other host. The domain has no site of its own; it points at one. That is a valid configuration when it is complete, and a defect when only the homepage redirects while robots.txt or the sitemap still answer on the alias.

Why is a canonical pointing at another page sometimes correct?

Paginated lists, print views and parameterised copies legitimately point at a parent page. The failure is a page reached through the site's own navigation, answering 200, whose canonical points elsewhere across a whole template, because that removes the section from the index while every page still answers.

How do query parameters create canonical problems?

A parameter that produces a distinct 200 page with a self-referencing canonical is a duplicate. The safe shapes are a redirect to the clean address, or the page served with a canonical to the clean address while internal links never carry the parameter.

Related findings

How anything measured in this article was measured15client identitiesone second, one address5machine filesapex and www114named agentsresolved from robots.txt24sections scoredreach, read, quoteHow anything measured here was measured15 client identities5 machine files114 named agents24 sections scoredone second, one addressapex and wwwresolved from robots.txtreach, read, quote
No account, nothing installed, and the same sequence on every domain — which is what makes one scan comparable to another. Run it on your own site.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works