CrawlCheck

Findings · 2026-09-16 · By · 0 views

Our crawl-waste check named a parameter called amp;plan. No site has one.

Nineteen parameter names in the corpus began with amp; because an href was split on & before its escaped ampersands were decoded. The counts were right the whole time. The names beside them were not.

On 16 September 2026 we read every crawl-waste record in the corpus. The section had been publishing parameter names that do not exist. Nineteen of the names it reported began with amp;: amp;utm_medium, amp;utm_campaign, amp;plan, amp;return, amp;aid. No site on the web has a query parameter called amp;plan. Our reader had manufactured them.

The cause is one line. An href in an HTML attribute carries its ampersands escaped, &, because a bare & starts an entity. Split the raw attribute value on & and the split lands on &'s own ampersand, leaving amp; glued to the front of every parameter after the first. A URL parser never complains, because amp;plan=pro is a perfectly legal key-value pair.

WHAT THE HTML CARRIEShref="/pricing?utm_source=footer&utm_medium=link&plan=pro"SPLIT ON & AS-IS (before)UNESCAPE FIRST (after)utm_sourceamp;utm_mediumamp;planutm_sourceutm_mediumplanThree parameters either way, so the count was right. Two of the three names never existed.

Why nothing caught it #

The count was right. A link with ?a=1&b=2 is one link with a query string whether you split it well or badly, so every number the section published was correct. The one field that was wrong, the most common parameter names printed under it, was the field a human reads to decide what to fix. A reader seeing amp;utm_medium would have gone looking for a parameter that was never there.

Our tests passed because their URLs were typed by hand with bare ampersands, which is how a person types a URL and not how a page serves one. The same class of defect has appeared here before, in the other direction: a FAQ check that compared schema text against rendered text and missed five matches because ' in the HTML was never unescaped before the comparison. Escaping is the boundary between what a page carries and what a page means, and every reader that crosses it without decoding produces answers that look like data.

The fix, and what it changed #

The attribute value is now decoded before the URL is parsed. One site that had read hsLang, utm_source, amp;utm_medium, amp;utm_campaign was rescanned the same hour and read hsLang, utm_source, utm_medium, utm_campaign. The counts on that record did not move: 133 of 183 internal links still carry a query string, a figure hand-checked against a separate fetch.

Records scanned before the fix keep the old names until they are rescanned. A stored reading describes the reader that produced it, so no stored figure was rewritten.

What the calibration found, once the names were right #

The section shipped on 9 September. The read on 16 September covered 111 records, every scan since then. The table is the share of those 111 carrying each signal.

SignalShare of 111 measured sitesReading
Internal links carrying a query string23%discriminates; the 90th percentile is 3% of links, the worst site 73%
Uppercase in an internal path13%discriminates
Declared sitemap URLs that redirect8%a true defect on every site it fires on
utm_ on internal links7%a true defect, rare
Apex and www both serving 2006%a true defect
Sitemap trailing slash used both ways5%marginal
Sitemap URLs with uppercase, a query, or http6%, 2%, 1%too rare to weight alone
Declared sitemap URLs that are gone2%too rare to weight alone
Internal links to http://3%too rare to weight alone
Any signal46%
Two or more18%

Measured on all 111 sites, the signals split the corpus 46 to 54, and the largest accusation in the set, the site with 133 query-string links, was checked by hand against an independent fetch.

Why the redirecting-sitemap row matters more than its 8% #

A declared URL that redirects is a fetch that returns nothing: the crawler spends a request learning the address it should have been given. It is also the row most often caused by something nobody considers a defect, a trailing-slash rule or an http to https upgrade applied after the sitemap generator ran. The sitemap audit guide covers the two-layer check that finds it. The apex-and-www row is the other one that hides in plain sight, and it is the defect this site carried for months with a correct canonical tag.

What this post does not claim #

It does not claim the shares above are stable; they are a reading of 111 records on one day and the live rates are on the dataset page. It does not claim the count was ever wrong, because it was not. It claims that a field printed beside a correct number was wrong for a week, that it was wrong in a way no test written by a person typing URLs would find, and that the fix is a decode step at the one boundary every HTML reader has to cross.

Every figure above came out of this scanner.

Point it at your own domain and see the same measurements, free.

Scan a domain — free

The main product

Found this on your own site? We fix it for $749.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Questions this post answers

Were the crawl-waste numbers wrong?

No. The share of internal links carrying a query string is counted per link, and a link has a query string whether its parameters are split correctly or not. Only the printed parameter names were wrong, on records scanned before the fix.

Why did amp; appear in front of the names?

HTML escapes ampersands inside attributes as &. Splitting the raw attribute value on the ampersand cuts inside that escape and leaves amp; attached to every parameter after the first. Decoding the value before parsing removes it.

Were the old records corrected?

No. A stored reading describes the reader that produced it, so the old names age out as sites are rescanned. Editing a past reading would make it something other than a reading.

Related findings

How anything measured in this article was measured15client identitiesone second, one address5machine filesapex and www114named agentsresolved from robots.txt24sections scoredreach, read, quoteHow anything measured here was measured15 client identities5 machine files114 named agents24 sections scoredone second, one addressapex and wwwresolved from robots.txtreach, read, quote
No account, nothing installed, and the same sequence on every domain — which is what makes one scan comparable to another. Run it on your own site.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works