Agent view · Findings · 2026-11-06

Most of our 'referring pages' were forged Referer headers

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/most-of-our-referring-pages-were-forged-referer-headers: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-11-06 · By [VSNARY | Emmanuel Orta](/author) · 3 views

# Most of our 'referring pages' were forged Referer headers

The owner panel that lists who links to a site was reporting its own IP addresses, parked lookalike domains and a radiology college. None of them had linked to anything. How a referrer is forged, and the rule that now counts one.

A Referer header is a claim, not a record. Until 6 November 2026 CrawlCheck counted every one it received as a referring page. Workers Logs showed the top entries were forgeries: crawlcheck.io's own Cloudflare addresses on ports 2082 and 2086, sent by a scanner on Google Cloud probing for control panels; search-engine lookalikes such as bing.io and google.io that do not resolve; and real organisations' hostnames with no page path. A referral now counts only when the request also carries the browser signals a real click produces: Sec-Fetch-Site cross-site, Sec-Fetch-Mode navigate and an Accept-Language header, on a GET.

On this page

- What was in the panel
- Why the forgeries exist
- The rule that now counts a referral
- How to check your own referrer report

The owner dashboard has a panel called referring pages. It is meant to answer a simple question: which outside pages are sending people here? On 6 November 2026 we read it against the raw request logs and found that most of what it listed had never linked to us at all.

## What was in the panel #

| Entry | What it really was | How we know |
| --- | --- | --- |
| 104.21.64.145:2082 and 172.67.151.148:2086 | crawlcheck.io's own Cloudflare addresses, with cPanel port numbers appended | Sent by two Google Cloud addresses using a Chrome 70 [user agent](/glossary/user-agent) while requesting /aab9, /aaa9 and /getcmd, which are control-panel probe paths |
| bing.io, google.io, yandex.io, corp.google.com | Lookalike search-engine hostnames | None resolve in DNS, or resolve to a parking page. A browser cannot arrive from a page that does not exist |
| storeopinion-ca.today, myfordbenefitsguide.co | Spam domains | Do not resolve |
| aocr.org | The American Osteopathic College of Radiology | 882 'visits' with no path at all. A real click carries the page it came from |
| toolgrep.com/p/crawlcheck, huggingface.co, kagi.com | Real | toolgrep lists CrawlCheck (35 mentions); Hugging Face hosts our datasets; Kagi is a search engine |

Two more entries were our own alternate domains, which redirect to us and are not referrals in any useful sense.

## Why the forgeries exist #

A Referer header is text the client chooses to send. Scanners set it for two reasons. Some set it to the target's own address to look like an internal navigation and slip past rules that trust same-site traffic. Some set it to a search engine so that a log reader sees 'traffic from Google' and does not look closer. The cPanel ports are the tell: CrawlCheck was looking for a hosting control panel on ports 2082 and 2086 and wrote that target into the header it sent along.

The radiology college is the stranger case. Its hostname arrived 882 times with no path. We did not find a page there that links to us, and a click from a page carries that page's [URL](/glossary/url). The likeliest explanation is a referrer-spam tool cycling through a list of real domains. Whatever the cause, nothing on that site sent us a visitor.

## The rule that now counts a referral #

A browser following a link from another site sends more than a Referer. It sends `Sec-Fetch-Site: cross-site`, `Sec-Fetch-Mode: navigate`, an `Accept-Language` header and uses GET. Scanners that forge a Referer almost never forge the fetch-metadata headers, because those are set by the browser and most tools do not know they exist.

A referral now counts only when all of those are present. In addition, a Referer that is a bare IP address is dropped, hosts confirmed forged were removed from the stored history, our own alternate domains are classified as our own, and Kagi was added to the list of search engines.

After the change the panel lists le-atelier-noir, nutritionome, toolgrep, a GreenGeeks help-desk article and Hugging Face. The engines list no longer includes google.io or yandex.io.

## How to check your own referrer report #

Most analytics tools apply some version of this filter already, which is why a Google Analytics referral list looks cleaner than a raw server log. If you read logs or run your own counter, three checks remove most of the noise: drop any Referer that is an IP address or your own host; drop any whose hostname does not resolve; and if your edge gives you fetch-metadata headers, require cross-site navigate. We had written about [how few real arrivals carry a referrer at all](/blog/ninety-three-percent-of-our-arrivals-sent-no-referrer), and the forged ones were a large share of the few that did.

It is not the first count of ours inflated by requests that only looked like the thing being counted. [Forged crawler user agents](/blog/how-much-ai-crawler-traffic-is-forged) did it to our crawler counts, and headless page loads did it to our engagement figures until this week. The pattern is the same each time: a header is a claim, and a claim is counted only once something the sender cannot easily fake agrees with it.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fmost-of-our-referring-pages-were-forged-referer-headers%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fmost-of-our-referring-pages-were-forged-referer-headers%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Most%20of%20our%20)[Email](mailto:?subject=Most%20of%20our%20)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Can a Referer header be faked?

Yes. It is plain text the client chooses to send. Scanners set it to the target's own address, to a search engine, or to a random real domain so that a log reader trusts the request or misreads where traffic came from.

### How do you tell a real referral from a forged one?

A browser following a link sends Sec-Fetch-Site cross-site, Sec-Fetch-Mode navigate and an Accept-Language header on a GET. Most tools that forge a Referer do not set those, because the browser sets them automatically.

### Why were CrawlCheck's own IP addresses in its referrer list?

A scanner on Google Cloud probed for a hosting control panel on ports 2082 and 2086 and wrote that target into the Referer it sent. CrawlCheck counted the header as a referring page.

### Did the American Osteopathic College of Radiology link to CrawlCheck?

No. Its hostname arrived 882 times with no page path, which a real click always carries. No page on that site links here. It is most likely a referrer-spam tool cycling through real domain names.

### What should I do about forged referrers in my own logs?

Drop Referer values that are IP addresses or your own host, drop hostnames that do not resolve, and if your edge exposes fetch-metadata headers, count a referral only when they show a cross-site navigation.

## Related findings

[2026-10-14 An access rule on one path locked our customers out An access rule for staff pages also covered customer sign-in at the same address. For an hour 247 pages linked to a login customer](/blog/cloudflare-access-on-one-path-locked-our-customers-out-of-sign-in)[2026-08-28 Free AI visibility checkers: how to tell if one is any good What free AI visibility checkers really test, why a 100% score can still come with a fix to buy, and four checks a free tool must](/guides/free-ai-visibility-checkers-what-they-measure)[2026-09-16 Canonical URL audit: do tags, redirects and sitemap agree? Audit canonicals from the responses, not the tag: apex and www both serving, half-done aliases, template canonicals and sitemaps t](/guides/canonical-url-audit-one-page-one-address)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [LLMs](/registry/llms) · [Agents](/registry/agents) · [augmentcode.com](/registry/augmentcode.com) · [crunchbase.com](/registry/crunchbase.com) · [fireworks.ai](/registry/fireworks.ai) · [inflection.ai](/registry/inflection.ai) · [mgvautomaticgearbox.ro](/registry/mgvautomaticgearbox.ro) | **Glossary:** [agent arrival](/glossary/agent-arrival) · [DefinedTermSet](/glossary/definedtermset) | **Directory:** [Weights](/directory/weights)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/most-of-our-referring-pages-were-forged-referer-headers.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.