Agent view · Findings · 2026-08-19
Is your auditor grading your firewall? A two-request test
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/the-site-that-scored-15-because-it-refused-us: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/the-site-that-scored-15-because-it-refused-us>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 14,627 |
| words | 2,111 |
| sha-256 | ecdcc865ae7cd0abd8c9f9830dada7a179c2a62d23949f79354707015f8b23f7 |
| repr-digest | sha-256=:7NzIZa580KvYyfmDDa2noXnCpi0jlJ95NUcHAV+LI/c=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311494;expires=1795315094;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="M-P6MtB79gyVgYk2TOV0gw";tag="crawlcheck-page" |
| signature | sig1=:0dongy/mG8jw34i05X1MDWKfR0YWnzwaeOqBYbQofGTUiPJbCqNh+CE9dGgCLbZXq/rhKo9LFMy+EsCYpeUSDg==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-08-19 · By [VSNARY | Emmanuel Orta](/author)
# Is your auditor grading your firewall? A two-request test
We once published a failing grade for a competitor that measured only its firewall. The random-path test that would have caught it, and the curl commands to run it on your own site.
If a site answers its homepage and a path that cannot exist with the same small page, a firewall is answering for it and nothing about the website has been measured. We once graded a competitor 15 out of 100 that way: its hosting firewall challenged our datacentre address on every path, while from a home connection its robots.txt and sitemap were normal. Running the scan twice repeated the error, because a block reproduces perfectly. Test any auditor with a random path and a second vantage point.
On this page
- How do I know my auditor measured my website?
- Why didn't running the scan twice catch it?
- When is a challenge page the site's problem?
- What changed
- What to ask any audit tool
Any tool that audits your site from cloud servers can be handed a firewall challenge page and grade it as your website. We know because we did it: for several hours CrawlCheck published a report grading a competitor 15 of 100, an F, with findings like "no sitemap" and "[robots.txt](/glossary/robots-txt) not served". All of it was false. From an ordinary home connection their robots.txt was a normal 251-byte file that declared a sitemap. What we had measured was their hosting firewall challenging our datacentre address on every path, at HTTP 200, in identical responses of about 200 bytes.
## How do I know my auditor measured my website? #
Ask it to fetch a path that cannot exist. A real website answers its homepage and a random path differently: a 404, or at least a different page. A site that returns the same small body with a 200 for both is not describing itself; something in front of it is speaking on its behalf. You can run the same test yourself, once from a cloud server or an online fetch tool and once from your own connection:
```
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" "https://example.com/no-such-page-$(date +%s)"
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/robots.txt
```
Each line prints the status code and the number of bytes. If the homepage and the random path come back with the same status and nearly the same size from the cloud server, but differ from your home connection, the cloud request is being challenged. Any grade built on that fetch describes your firewall.
The tell that a scan measured a firewall, not a website.
## Why didn't running the scan twice catch it? #
We re-ran the scan and got the same 15. That felt like confirmation and was not: a block reproduces perfectly, so two identical readings from one [vantage point](/glossary/vantage-point) are one reading. Repetition tests for flukes. It does not test whether you were looking at the right thing. What settles it is a second vantage point, or the random-path control above.
## When is a challenge page the site's problem? #
The same bytes mean different things depending on where they turn up, and collapsing them is how the 15 happened.
| What the fetch received | What it means | Whose problem |
| --- | --- | --- |
| Homepage and a random path answer with the same challenge body | The edge is refusing this address, not describing the site | The auditor's. Nothing about the site was measured. |
| Homepage is real; `robots.txt` is a [challenge page at 200](/glossary/challenge-page-at-200) | Machine files sit behind a wall the page does not. Real crawlers get this too. | The site's, because any crawler receives it. |
| That challenge page carries `cf-cache-status: HIT` | A temporary challenge was cached and is now served to everyone | The site's, and more serious: the challenge is pinned at the edge. |
| One crawler name is refused, the browser is not | A policy, not breakage | Neither: a deliberate block is a choice. Check it is the one you meant. |
The second row is the one people expect to be unfair and is not. [A robots.txt that answers 200 with a verification interstitial](/blog/a-file-that-answered-200-and-could-not-be-read) is served that way to Googlebot as well, and a parser reads it as an empty policy, which means everything allowed. The first row is the one that was unfair.
## What changed #
We deleted the false report and the record behind it. A scan that meets a site answering for every path the same way now reports that the site refused measurement and gives no grade. A check that could not be verified no longer counts against a site, and the report says which checks could not be verified. A [refusal](/glossary/refusal) of our address to one crawler name is reported as a refusal of us, not as proof of what the real crawler gets; [what each crawler receives](/blog/what-a-crawler-actually-receives) is measured on that basis.
The [vendor directory](/directory) shows this in public: a tool whose edge refused CrawlCheck is listed as not graded, with no number, and is not marked down. A vendor that challenges datacentre traffic has made a defensible choice, and a scanner that punished it would be measuring its own address.
## What to ask any audit tool #
One refusal is also not a block: [a single 503 can be a hiccup](/blog/one-503-is-not-a-blocked-site-how-to-tell-a-refusal-from-a-hiccup), and a report should say how often it saw a refusal, not just that it saw one. Before acting on any audit that says your sitemap is missing or your robots.txt is broken, ask: how do you know what you measured was my website? If the answer does not involve a second vantage point or a control like a random path, run the commands above yourself.
**Live, as you read this:** the corpus now holds 582,334 domains across 9,306 scans. The figures in this piece were measured on the date above; this line is not.
CrawlCheck’s dataset when this page loaded: 582,334 sites measured.
Written by
[Emmanuel Orta](/author) · VSNARY
I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.
[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)
Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-site-that-scored-15-because-it-refused-us%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-site-that-scored-15-because-it-refused-us%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Is%20your%20auditor%20grading%20your%20firewall%3F%20A%20two-request%20test)[Email](mailto:?subject=Is%20your%20auditor%20grading%20your%20firewall%3F%20A%20two-request%20test&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-site-that-scored-15-because-it-refused-us)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)
## Questions this post answers
### Can an SEO or AI visibility tool grade a firewall instead of a website?
Yes. If the site's firewall challenges the tool's cloud address, every check reads the challenge page, and the resulting grade describes the firewall, not the site.
### How can I tell whether a scan measured my site or my firewall?
Fetch your homepage and a random path that cannot exist from a cloud server. If both return the same status and nearly the same small size, something in front of the site is answering. Compare with the same fetch from your own connection.
### Why does running a scan twice not prove it is right?
A block reproduces perfectly, so two identical readings from the same vantage point are one reading. A second vantage point or a control request is what tests whether the right thing was measured.
### Is a challenge page on robots.txt the site's fault?
Yes, when the homepage is served normally but robots.txt returns a challenge page. Real crawlers receive that page too and read it as an empty policy, so it is a real defect on the site.
### What does CrawlCheck do when a site refuses it?
It reports that the site refused measurement and gives no grade. A refusal of its address is not counted as a defect of the site.
## Related findings
[2026-08-28 Free AI visibility checkers: how to tell if one is any good What free AI visibility checkers really test, why a 100% score can still come with a fix to buy, and four checks a free tool must](/guides/free-ai-visibility-checkers-what-they-measure)[2026-08-28 Does llms.txt work? What is known and how to test it Does llms.txt work? What is verifiable, what we measured on our own site, and the server-log test with IP verification that answer](/guides/does-llms-txt-actually-work)[2026-09-16 Canonical URL audit: do tags, redirects and sitemap agree? Audit canonicals from the responses, not the tag: apex and www both serving, half-done aliases, template canonicals and sitemaps t](/guides/canonical-url-audit-one-page-one-address)
Where this is used
Guides that cite this finding:
- [Free AI visibility checkers: how to tell if one is any good](/guides/free-ai-visibility-checkers-what-they-measure)
- [How to check if AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site)
- [How to measure AI visibility yourself: three checks](/guides/how-to-measure-ai-visibility)
The main product
Found this on your own site? **We fix it for $100.**
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)
Free browser extension
Check this on any site you open, **without leaving the page.**
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)
Write for the library
Measured something yourself? **Submit an article.**
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week
Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).
## Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.
Send for review
Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.
[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)
We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)
#### How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).
**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.
**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.
**Pages:** [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [SDK](/sdk) | **Registry:** [autonationchevroletclearwater.com](/registry/autonationchevroletclearwater.com) · [denverhomeservicesmasters.com](/registry/denverhomeservicesmasters.com) · [hurriyet.com.tr](/registry/hurriyet.com.tr) · [nevian.com](/registry/nevian.com) · [tiktok.com](/registry/tiktok.com) · [Trade: Dentist](/registry/trade/dentist) | **Datasets:** [Utah](/datasets/state/utah) · [Technologies: Next js](/datasets/technologies/next-js)
[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/the-site-that-scored-15-because-it-refused-us.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.