CrawlCheck

Findings · 2026-08-19 · By

Is your auditor grading your firewall? A two-request test

We once published a failing grade for a competitor that measured only its firewall. The random-path test that would have caught it, and the curl commands to run it on your own site.

If a site answers its homepage and a path that cannot exist with the same small page, a firewall is answering for it and nothing about the website has been measured. We once graded a competitor 15 out of 100 that way: its hosting firewall challenged our datacentre address on every path, while from a home connection its robots.txt and sitemap were normal. Running the scan twice repeated the error, because a block reproduces perfectly. Test any auditor with a random path and a second vantage point.

Any tool that audits your site from cloud servers can be handed a firewall challenge page and grade it as your website. We know because we did it: for several hours CrawlCheck published a report grading a competitor 15 of 100, an F, with findings like "no sitemap" and "robots.txt not served". All of it was false. From an ordinary home connection their robots.txt was a normal 251-byte file that declared a sitemap. What we had measured was their hosting firewall challenging our datacentre address on every path, at HTTP 200, in identical responses of about 200 bytes.

How do I know my auditor measured my website? #

Ask it to fetch a path that cannot exist. A real website answers its homepage and a random path differently: a 404, or at least a different page. A site that returns the same small body with a 200 for both is not describing itself; something in front of it is speaking on its behalf. You can run the same test yourself, once from a cloud server or an online fetch tool and once from your own connection:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" "https://example.com/no-such-page-$(date +%s)"
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/robots.txt

Each line prints the status code and the number of bytes. If the homepage and the random path come back with the same status and nearly the same size from the cloud server, but differ from your home connection, the cloud request is being challenged. Any grade built on that fetch describes your firewall.

GET /GET /random-path200, ~200 bytes200, ~200 bytesidentical:a firewall isansweringA real site answers a path that cannot exist with a 404 or a different page, never with its homepage bytes.
The tell that a scan measured a firewall, not a website.

Why didn't running the scan twice catch it? #

We re-ran the scan and got the same 15. That felt like confirmation and was not: a block reproduces perfectly, so two identical readings from one vantage point are one reading. Repetition tests for flukes. It does not test whether you were looking at the right thing. What settles it is a second vantage point, or the random-path control above.

When is a challenge page the site's problem? #

The same bytes mean different things depending on where they turn up, and collapsing them is how the 15 happened.

What the fetch receivedWhat it meansWhose problem
Homepage and a random path answer with the same challenge bodyThe edge is refusing this address, not describing the siteThe auditor's. Nothing about the site was measured.
Homepage is real; robots.txt is a challenge page at 200Machine files sit behind a wall the page does not. Real crawlers get this too.The site's, because any crawler receives it.
That challenge page carries cf-cache-status: HITA temporary challenge was cached and is now served to everyoneThe site's, and more serious: the challenge is pinned at the edge.
One crawler name is refused, the browser is notA policy, not breakageNeither: a deliberate block is a choice. Check it is the one you meant.

The second row is the one people expect to be unfair and is not. A robots.txt that answers 200 with a verification interstitial is served that way to Googlebot as well, and a parser reads it as an empty policy, which means everything allowed. The first row is the one that was unfair.

What changed #

We deleted the false report and the record behind it. A scan that meets a site answering for every path the same way now reports that the site refused measurement and gives no grade. A check that could not be verified no longer counts against a site, and the report says which checks could not be verified. A refusal of our address to one crawler name is reported as a refusal of us, not as proof of what the real crawler gets; what each crawler receives is measured on that basis.

The vendor directory shows this in public: a tool whose edge refused CrawlCheck is listed as not graded, with no number, and is not marked down. A vendor that challenges datacentre traffic has made a defensible choice, and a scanner that punished it would be measuring its own address.

What to ask any audit tool #

One refusal is also not a block: a single 503 can be a hiccup, and a report should say how often it saw a refusal, not just that it saw one. Before acting on any audit that says your sitemap is missing or your robots.txt is broken, ask: how do you know what you measured was my website? If the answer does not involve a second vantage point or a control like a random path, run the commands above yourself.

Live, as you read this: the corpus now holds 582,334 domains across 9,304 scans. The figures in this piece were measured on the date above; this line is not.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Can an SEO or AI visibility tool grade a firewall instead of a website?

Yes. If the site's firewall challenges the tool's cloud address, every check reads the challenge page, and the resulting grade describes the firewall, not the site.

How can I tell whether a scan measured my site or my firewall?

Fetch your homepage and a random path that cannot exist from a cloud server. If both return the same status and nearly the same small size, something in front of the site is answering. Compare with the same fetch from your own connection.

Why does running a scan twice not prove it is right?

A block reproduces perfectly, so two identical readings from the same vantage point are one reading. A second vantage point or a control request is what tests whether the right thing was measured.

Is a challenge page on robots.txt the site's fault?

Yes, when the homepage is served normally but robots.txt returns a challenge page. Real crawlers receive that page too and read it as an empty policy, so it is a real defect on the site.

What does CrawlCheck do when a site refuses it?

It reports that the site refused measurement and gives no grade. A refusal of its address is not counted as a defect of the site.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works