Agent view · Findings · 2026-08-19
Is your host blocking ClaudeBot? Group the 403s by IP first
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/we-almost-published-that-our-host-was-blocking-claude: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/we-almost-published-that-our-host-was-blocking-claude>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 19,237 |
| words | 2,819 |
| sha-256 | 5f42d30985d81152ffd743323095d54a65a1ed74c6b0f09d81e78c501aae307a |
| repr-digest | sha-256=:X0LTCYXYEVL/10MyMJXVSmWh7XTGsPCdgeeMUBquMHo=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311490;expires=1795315090;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="43QmC3lEa-arnZ3DxfnflA";tag="crawlcheck-page" |
| signature | sig1=:YywAMlmKc1gI5SmiJXulk5vEa0A2tY8ZsCw3V6NGnbkz2sl9nfdZNHJmAsHXYWjVOLIb0PP2krjuy8GKwIp5AQ==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3599 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-08-19 · By [VSNARY | Emmanuel Orta](/author) # Is your host blocking ClaudeBot? Group the 403s by IP first A 57% block rate in our own access logs looked like a hosting provider shutting out AI crawlers. Grouped by source address, almost all of it was one machine wearing seven different crawler names. We were one edit from publishing that a managed host was blocking Claude across five applications. Grouped by user-agent, the refusals looked like seven crawler operators being turned away. Grouped by source address, they were one machine sending seven different crawler names — which means at least six were forged, and the refusals were a defence working correctly. A smaller set was real, concentrated on the machine files, and was fixed by putting those endpoints behind an edge cache. Measured across nine domains on 19 August, eighteen of eighteen requests returned 200. On this page - The part where we nearly got it wrong - The refusals that did matter - Where it stands today - Why one machine looked like seven companies - The layer underneath: those names were claims, not identities - What the genuine refusals had in common - The publishing rule this produced - What to do with your own logs Share of ClaudeBot requests refused, two sites on the same host First site, July 553 refused of 642 First site, August 23 refused of 171 Second site, July 136 refused, 29 served Second site, August 15 refused, 117 served In July, AI crawlers were being refused across five WordPress applications we run on one managed host. This is not a guess from a sample: it is in the access logs. On one site `ClaudeBot` received **553 refusals out of 642 requests** that month. On another, **136 refusals against 29 successes** — 82% of everything Anthropic's crawler asked for. On 30 July that stopped. The August logs for the same two sites read **23 refusals out of 171**, and **117 successes against 15 refusals**. The change lands on one date across applications that share nothing but a host, which is the signature of a platform default rather than anything in our configuration. We never saw the rule itself and cannot show you its text — what we can show is five applications changing behaviour on the same day, in logs we did not write. | ClaudeBot on two sites, same host | July | August | | --- | --- | --- | | First site | 553 refused of 642 | 23 refused of 171 | | Second site | 136 refused, 29 served | 15 refused, 117 served | ## The part where we nearly got it wrong # The refusals did not go to zero. On one site a single day in August showed **303 crawler requests and 172 refusals** — 57%. Written down that way it reads like the block never really lifted, and that sentence was close to being published. Then we grouped the refusals by source address. **All 172 came from one IP.** That address presented **seven different crawler identities** in a single day: Amazonbot, ChatGPT-User, ClaudeBot, OAI-SearchBot, GPTBot, [Google-Extended](/glossary/google-extended) and PerplexityBot. It sits in no range any of those operators publishes. On a second site the same shape appeared — 151 of roughly 229 refusals from one Google Cloud address, again wearing seven names. One host cannot be seven companies. What the 57% actually measured was a scraper being correctly refused, and counting it as blocked crawler traffic did not just inflate the number — it inverted the conclusion. The genuine crawlers were fine in the same logs: **Googlebot refused zero times, Bingbot zero.** ## The refusals that did matter # A smaller set was real, and it was concentrated in the worst possible place. Of just over a thousand AI-crawler refusals on one site, **316 were the sitemap** and **24 were /robots.txt**. A crawler that cannot read your sitemap cannot enumerate your site and falls back to following links. Worse, most compliant crawlers treat a [refusal](/glossary/refusal) on `robots.txt` as *disallow everything* — not as *no rules found*. A few dozen refusals on one small file explains more lost coverage than any amount of theorising about crawl demand. [What each robots.txt status code tells a crawler](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler) covers the rule in detail. Those were fixed by putting the machine files behind an [edge cache](/glossary/edge-cache), so the high-frequency endpoints stop touching the origin at all. The remaining refusals are vulnerability probes — requests for `/.ssh/`, `/.claude.json`, `/.env` — where refusing is the correct behaviour and unblocking would be a mistake. ## Where it stands today # Measured on 19 August 2026, across nine domains, requesting the homepage and `/robots.txt` as `ClaudeBot` and as a browser: **eighteen of eighteen returned HTTP 200**, with no challenge page in any response body. Nothing on this network refuses Claude today. ## Why one machine looked like seven companies # The chart that nearly became a headline was refusals grouped by user-agent. That grouping is the error, and it is an easy one to make because every log tool offers it and the resulting bar chart looks like a finding. A user-agent is a field in the request. Grouping by it counts *names sent*, not *senders*, and those are the same number only if every sender uses one name. Regroup the identical rows by source address and the seven bars collapse into one, because they were one client cycling through identities. The general form is worth keeping: **a rate is only a rate if its denominator counts independent events.** Seven refusals to one address inside a minute is one event observed seven times. Presented as seven, it supports a sentence about seven operators that never happened. This is the same arithmetic that makes small-sample forgery rates useless: without a minimum sample, the loudest single misbehaving client sets the headline. ## The layer underneath: those names were claims, not identities # Once the rows collapse to one address, a second fact becomes unavoidable. A single machine cannot be seven crawler operators. At most one of those names was true, so at least six were forged — and both large operators involved publish address ranges, which means the claims were checkable and failed. [How much AI crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged) runs the same check across every named crawler. That inverts the story completely. The chart was not evidence of a host blocking AI crawlers. It was evidence of something impersonating AI crawlers and being correctly refused, which is the defence doing exactly what it exists to do. Had we published, we would have reported a working security control as an outage, on a host we were about to name. It also explains why the refusal count did not fall to zero when the real problem was fixed on 30 July, and why we should have treated that residue as a question rather than as a remainder. A number that stops falling when the cause is removed is telling you there was more than one cause. ## What the genuine refusals had in common # The smaller, real set was not distributed across the sites. It was concentrated almost entirely on the machine files — `robots.txt`, `llms.txt`, the sitemap — and the reason is mechanical rather than sinister. Those endpoints are the highest-frequency and lowest-cost requests any crawler makes. A well-behaved operator re-reads [robots.txt](/glossary/robots-txt) often, by design. So on a per-address rate limiter they are the first requests to cross a threshold, and they are the requests whose refusal does the most damage, because a crawler that cannot read your policy file either stops or proceeds without it. Why cache the files rather than allowlist the crawlers? An allowlist keyed on a user-agent is unenforceable, for the reason the section above demonstrates in our own data. Caching the file removes the origin from the path regardless of who is asking, and it is enforceable because it does not depend on the request being honest about itself. If you cache machine files, check the cached copy stays current: [stale cached robots.txt and llms.txt](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge) covers how. ## The publishing rule this produced # We had a headline available in July and it would have been true in July. It stopped being true on 30 July, and the version we nearly published in August was a story about a host that had already fixed the problem, inflated by a forgery artefact we had not checked for. The rule now: **any finding that accuses a named third party gets re-verified by hand, on the host CrawlCheck actually read, immediately before publication.** Not on the domain you typed — on the host the request resolved to, which is a different thing whenever an apex redirects to `www`, and has caught a near-miss here for exactly that reason. The cost is that some true stories go out late, and one or two die because they stopped being true while we were checking. That is the correct trade for a site whose proposition is that claims should be checkable. A measurement business that publishes a wrong accusation has not made a small error; it has disproved its own product. ## What to do with your own logs # If you are about to conclude that something is blocking AI crawlers on your site, do these three things first, in this order: **Group refusals by source IP.** One address responsible for most of them means a scraper, not a policy. **Count identities per address.** Any single IP claiming more than one named crawler is forging at least one of them. **Check the paths.** Refusals on `robots.txt` and your sitemap cost you real coverage; refusals on credential paths are your server doing its job. A blocked-rate with none of that behind it is not a measurement. It is a number that happens to be large, and large numbers travel further than correct ones. **Live, as you read this:** the corpus now holds 582,334 domains across 9,306 scans. The figures in this piece were measured on the date above; this line is not. CrawlCheck’s dataset when this page loaded: 582,334 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwe-almost-published-that-our-host-was-blocking-claude%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwe-almost-published-that-our-host-was-blocking-claude%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Is%20your%20host%20blocking%20ClaudeBot%3F%20Group%20the%20403s%20by%20IP%20first)[Email](mailto:?subject=Is%20your%20host%20blocking%20ClaudeBot%3F%20Group%20the%20403s%20by%20IP%20first&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fwe-almost-published-that-our-host-was-blocking-claude)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Is my host blocking ClaudeBot? Group your 403s by source IP before answering. A high block rate can be one machine wearing several crawler names rather than a hosting policy. ### What did the grouping reveal here? Almost all of the apparent block came from a single address presenting as seven different crawlers. The host was not the cause. ### Which refusals did matter? The ones on robots.txt and the sitemap. A 403 on robots.txt reads to a parser as disallow everything. ### How do I tell if my host is blocking AI crawlers? Request your homepage and your robots.txt from outside your network, once as the crawler identity and once as an ordinary browser, and compare the status and the body of the final response. A refusal that appears for the crawler and not the browser is a block. Anything you conclude from a log chart alone, without that paired request, is an inference about a field the sender controls. ### Why did grouping refusals by user-agent give the wrong answer? Because a user-agent counts names sent rather than senders. One client cycling through seven crawler names produces seven bars and one actual source. Regrouping the same rows by source address collapsed them into a single machine, which also meant at least six of the seven names were forged. ### What should I check before concluding a crawler is being blocked? Three things, in order. Group the refusals by source address rather than by user-agent. Check whether those addresses fall inside the ranges the named operator publishes, where the operator publishes any. Then reproduce the refusal yourself with a direct request from outside your network. A blocked rate with none of that behind it is not a measurement. ## Related findings [2026-10-06 Only 6% of websites block an AI crawler: the full breakdown 6.3% of 139,788 homepages block an AI crawler in robots.txt. GPTBot leads at 3.7%; search agents are blocked far less. The full pe](/blog/only-six-percent-of-websites-block-an-ai-crawler)[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[2026-08-18 Which AI crawlers can't be verified by IP range? Which crawler operators publish IP ranges and which publish nothing, why an unverifiable claim is not a forgery, and what you can](/blog/the-crawlers-nobody-can-check) Where this is used Guides that cite this finding: - [Do AI crawlers and agents respect robots.txt?](/guides/do-ai-crawlers-respect-robots-txt) - [How to block AI crawlers like GPTBot and ClaudeBot](/guides/how-to-block-ai-crawlers) - [Is Cloudflare blocking AI crawlers on your site?](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [Agentapi](/registry/agentapi) · [coconutcreekchevrolet.com](/registry/coconutcreekchevrolet.com) · [example-shop.com](/registry/example-shop.com) · [lakewoodplumbingpros.com](/registry/lakewoodplumbingpros.com) · [rosstree.net](/registry/rosstree.net) · [Trade: Fencing](/registry/trade/fencing) · [exa.ai](/registry/capabilities/exa.ai) | **Pages:** [IP ranges](/data/ip-ranges) | **Glossary:** [Retry-After](/glossary/retry-after) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/we-almost-published-that-our-host-was-blocking-claude.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.