Findings · 2026-08-29 · By VSNARY | Emmanuel Orta
Our canonical tag was correct and the site was still served twice
Every page on this site existed at two addresses for months. The canonical pointed at the right one. That turns out not to be the part that matters.
An external audit of this site came back with a Health Score of 60 and four 404s. Two of the URLs were paths that do not exist for an anonymous visitor. The other two were the same two paths again, on a different hostname.
That second detail was the real finding. www.crawlcheck.io was answering 200 for every page on the site, serving a full copy alongside the apex. It had been doing that for as long as the domain has been live.
Why it looked fine
Because on the measure everyone checks, it was fine. Every page carried a correct canonical tag pointing at the apex:
| Request | https://www.crawlcheck.io/guides |
| Status | 200 |
| Canonical | https://crawlcheck.io/guides |
Nothing in that is wrong. The canonical is accurate, it points where it should, a validator passes it. And it does not solve the problem, because a canonical tag is read after the response arrives. The crawler resolved the hostname, opened a connection, and received the full page. The tag then told it which address to prefer. The second fetch had already happened.
Sixty-five pages, two hosts. Every crawl of this site cost twice what it should, and half of the requests were spent confirming that the other half were the ones that counted.
Why it survives
Three reasons, and they compound.
A browser never shows you. You type the domain without the prefix, you get the site, everything works. Nobody visits their own site by typing www. in front of it.
The tooling reports it as correct. A canonical checker looks at the tag and finds the tag is right. It is not asking the prior question, which is whether the request should have been served at all.
The default is to serve, not to redirect. Point a wildcard record or a proxied hostname at an origin and both names answer. Nothing has to be misconfigured for this to happen; it is what you get when nobody decides otherwise.
The fix
One redirect, before routing:
| Before | www.crawlcheck.io/guides → 200, full page |
| After | www.crawlcheck.io/guides → 301 → crawlcheck.io/guides |
Verified on the root, /blog, /guides and /docs/api. One address per page.
The direction does not matter. Redirecting the apex to www is equally correct. What matters is that one of the two stops answering with content.
What we added to the scanner
This is exactly the class of defect the scanner exists for: valid markup, healthy status codes, correct configuration by every local measure, and a crawler doing twice the work. So it is now a check.
HOST_DUPLICATE_200, severity 3. The scanner requests the peer hostname, follows redirects, and compares where it landed. A host that redirects lands back on the apex and passes silently. A host that answers 200 and stays there, with a body within 15% of the apex, is a second copy and gets reported.
Two constraints worth stating, because they decide whether the check is honest.
It only runs when the apex was the target. If someone deliberately scans a www hostname, the duplicate is the address they asked about, and reporting their choice back to them as a defect would be noise.
It compares bodies, not just status. A www host answering 200 with something genuinely different — a landing page, a legacy app, a redirect to a different product — is not this defect. Requiring the payloads to be within 15% of each other keeps those out.
Two results from the first runs
Pointed at real sites, with the peer host confirmed by hand first:
| Domain | Apex | www | Drift | Fired |
|---|---|---|---|---|
| ahrefs.com | 200 | 301 to apex | — | no |
| sqlite.org | 8,886 bytes | 8,886 bytes, stays on www | 0% | yes |
| apache.org | 54,021 bytes | 54,021 bytes, stays on www | 0% | yes |
Byte-identical on both, in both cases. These are not obscure sites run by people who do not know what a redirect is — they are among the most linked-to domains on the web. The defect is not a competence problem. It is a default nobody revisited.
The uncomfortable part
We shipped a scanner that reports what a machine actually receives, and we were serving our own site twice to every machine that asked. It went unnoticed because we checked the thing everyone checks — is the canonical correct — instead of the thing that decides the outcome, which is how many times the site answers.
The check exists now because we failed it. That is the only reason any of these checks exist.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Does a canonical tag stop duplicate content across www and the apex?
No. A canonical tag is read after the response arrives, so the crawler has already resolved the hostname, opened a connection and received the full page. The tag expresses a preference about which address to index; it does not prevent the second fetch.
How do I check whether my site is served on two hostnames?
Request your homepage on both the apex and the www hostname and compare the final URL after redirects. If both return 200 and the www request is still on www at the end, both hosts are serving the site. Byte counts within a few percent of each other confirm they are the same pages.
Should www redirect to the apex or the other way round?
Either is correct. What matters is that only one of the two answers with content and the other issues a 301 to it, so every page has a single address.