Findings · 2026-08-19 · By VSNARY | Emmanuel Orta
Is your host blocking ClaudeBot? Group the 403s by IP before you answer
A 57% block rate in our own access logs looked like a hosting provider shutting out AI crawlers. Grouped by source address, almost all of it was one machine wearing seven different crawler names.
In July, AI crawlers were being refused across five WordPress applications we run on one managed host. This is not a guess from a sample: it is in the access logs. On one site ClaudeBot received 553 refusals out of 642 requests that month. On another, 136 refusals against 29 successes — 82% of everything Anthropic's crawler asked for.
On 30 July that stopped. The August logs for the same two sites read 23 refusals out of 171, and 117 successes against 15 refusals. The change lands on one date across applications that share nothing but a host, which is the signature of a platform default rather than anything in our configuration. The host is Cloudways. We never saw the rule itself and cannot show you its text — what we can show is five applications changing behaviour on the same day, in logs we did not write.
The part where we nearly got it wrong
The refusals did not go to zero. On one site a single day in August showed 303 crawler requests and 172 refusals — 57%. Written down that way it reads like the block never really lifted, and that sentence was close to being published.
Then we grouped the refusals by source address. All 172 came from one IP. That address presented seven different crawler identities in a single day: Amazonbot, ChatGPT-User, ClaudeBot, OAI-SearchBot, GPTBot, Google-Extended and PerplexityBot. It sits in no range any of those operators publishes. On a second site the same shape appeared — 151 of roughly 229 refusals from one Google Cloud address, again wearing seven names.
One host cannot be seven companies. What the 57% actually measured was a scraper being correctly refused, and counting it as blocked crawler traffic did not just inflate the number — it inverted the conclusion. The genuine crawlers were fine in the same logs: Googlebot refused zero times, Bingbot zero.
The refusals that did matter
A smaller set was real, and it was concentrated in the worst possible place. Of just over a thousand AI-crawler refusals on one site, 316 were the sitemap and 24 were /robots.txt. A crawler that cannot read your sitemap cannot enumerate your site and falls back to following links. Worse, most compliant crawlers treat a refusal on robots.txt as disallow everything — not as no rules found. A few dozen refusals on one small file explains more lost coverage than any amount of theorising about crawl demand.
Those were fixed by putting the machine files behind an edge cache, so the high-frequency endpoints stop touching the origin at all. The remaining refusals are vulnerability probes — requests for /.ssh/, /.claude.json, /.env — where refusing is the correct behaviour and unblocking would be a mistake.
Where it stands today
Measured on 19 August 2026, across nine domains, requesting the homepage and /robots.txt as ClaudeBot and as a browser: eighteen of eighteen returned HTTP 200, with no challenge page in any response body. Nothing on this network refuses Claude today.
We had a headline available in July and it would have been true then. Published this week it would have been false, about a named company, on stale data. The check that makes it a story rather than an accusation is the boring one: re-run the measurement before you publish it.
What to do with your own logs
If you are about to conclude that something is blocking AI crawlers on your site, do these three things first, in this order:
Group refusals by source IP. One address responsible for most of them means a scraper, not a policy. Count identities per address. Any single IP claiming more than one named crawler is forging at least one of them. Check the paths. Refusals on robots.txt and your sitemap cost you real coverage; refusals on credential paths are your server doing its job.
A blocked-rate with none of that behind it is not a measurement. It is a number that happens to be large, and large numbers travel further than correct ones.
Live, as you read this: the corpus now holds 98 domains across 776 scans. The figures in this piece were measured on the date above; this line is not.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Is my host blocking ClaudeBot?
Group your 403s by source IP before answering. A high block rate can be one machine wearing several crawler names rather than a hosting policy.
What did the grouping reveal here?
Almost all of the apparent block came from a single address presenting as seven different crawlers. The host was not the cause.
Which refusals did matter?
The ones on robots.txt and the sitemap. A 403 on robots.txt reads to a parser as disallow everything.