CrawlCheck

Exclusive finding · Findings · 2026-08-18 · By · 1 view

Fake GPTBot and ClaudeBot requests scan for .env files

Of 874 recorded requests from clients claiming to be a named AI crawler, 69 asked for a file that holds secrets, such as cloud credentials, Terraform state and .env files, across at least 66 paths no real crawler requests.

Some requests that claim to be GPTBot, ClaudeBot or other AI crawlers are vulnerability scans. On this site, on 18 August 2026, 69 of 874 recorded requests carrying a named AI crawler's user agent asked for files that hold secrets, such as .env files, cloud credentials and Terraform state, across at least 66 paths. No real search or AI crawler requests those files. Verify a crawler by its IP address against the operator's published ranges, never by its name, and make sure secret files are never served from the web root.

Measured 18 August 2026
Requests claiming a named AI crawler874
Requests for files that hold secrets69

A forged user-agent is usually discussed as a statistics problem: someone inflating a crawl budget, or a scraper hiding behind a familiar name. On this site it is also a reconnaissance tool.

What arrived #

Measured 18 August 2026. Of 874 recorded path requests from clients claiming a named AI crawler, 69 were requests for files that hold secrets, spread across at least 66 distinct paths. That is a floor rather than a count: not every path was recorded, so the true number is higher. A sample, reproduced exactly as received:

  • /@fs/root/terraform.tfstate
  • /@fs/usr/src/app/.aws/credentials
  • /@fs/home/ubuntu/.env
  • /@fs/root/.aws/sso/cache/
  • /@fs/root/.config/hcloud/cli.toml
  • /@fs/..%252f..%252f..%252f..%252fapp/.env

The /@fs/ prefix is the giveaway. It is a path served by a popular JavaScript development server, and a misconfigured deployment will happily read files outside the project root through it. The doubled URL-encoding in the last example is a traversal attempt dressed to survive one round of decoding. No search or AI crawler has ever requested a Terraform state file.

How a claim is adjudicated, so you can run the same check #

Every request that carries a named crawler’s user-agent is one of three things and the record keeps them apart. Verified: the connecting address is inside the ranges the operator publishes for that name. Forged: the operator publishes ranges and the address is not in them. Unverifiable: the operator publishes nothing to check against, so the claim is neither confirmed nor accused. The third class is excluded from every forgery rate on this site rather than folded into either side, because a rate that counts the unknowable as fraud is the same defect as a scanner that scores a check that does not apply.

Name claimedChecked againstOutcome when the address is elsewhere
GPTBot, OAI-SearchBot, ChatGPT-UserOpenAI’s published JSON range files, one per agentforged
ClaudeBot, Claude-User, Claude-SearchBotAnthropic’s published rangesforged
Googlebot, Google-ExtendedGoogle’s published crawler rangesforged
An agent whose operator publishes no rangesnothingunverifiable — never counted as forged

You can run the same check on your own log lines with the free log check on CrawlCheck’s tools page, and the first-party tally of what reached this site is public on the live dataset, split into verified, forged and unverifiable.

Two rules that keep the rate honest #

If you compute a forgery rate from your own logs, two rules keep it honest. First, do not quote a per-crawler rate from a handful of claims: one spoofed request out of one is a 100% rate pretending to be a pattern. Second, count failed checks from your own tools and monitoring separately, so your own staff or customers checking a page are not reported as impersonators.

The one identity where attribution is unambiguous #

Where an identity has both verified and failed traffic, we cannot say which specific request came from whom — and we will not pretend otherwise.

One case has no such ambiguity. Every request claiming Google-Extended failed verification: 10 claims, none from Google's published ranges. Nine of the ten paths those requests asked for were credential files — every one except /: /.env, /production/.env, /@fs/.env.development, /@fs/var/www/.env, /@fs/root/.config/gcloud/application_default_credentials.json, /.aws/config, /.codex/config.toml, /openai.json and /private-key. Google-Extended is the identity a site owner uses to control whether their content trains Google's models — a name chosen, presumably, because it is one a cautious operator is least likely to block.

What 874 crawler-claiming requests actually asked for #

Measured 18 August 2026Count
Requests claiming a named AI crawler874
Requests for files that hold secrets69
Distinct secret-file pathsat least 66
Google-Extended claims that verified against Google’s published ranges0 of 10

The path count is a floor, not a total: not every path was recorded.

What this changes about blocking #

The usual advice is to decide which AI crawlers you allow and write that into robots.txt. That advice is sound and it is also inert against this: robots.txt is a request, honoured by the crawlers that were never your problem. The client asking for your .env is not reading it.

What does work is the same check that produces every figure on this site: adjudicate the name against the operator’s published address ranges, and treat a failure as untrusted traffic rather than as a crawler. That is a rule you can enforce at the edge in one place, and unlike a robots directive, it does not depend on the other party’s goodwill. How to block by range rather than by name is its own guide; the short version is that a name is a string anyone can type and an address range is not.

Why a 403 here is correct, and a robots line is not #

A 403 to a forged claim costs a real crawler nothing, because a real crawler never fails the check. A robots.txt disallow costs the real crawler its access and costs the forger nothing, because the forger does not read the file. So the two instruments point in opposite directions: the directive governs the well-behaved and the range check governs everyone else. The live dataset carries the running count of both kinds of traffic against this domain, so the sample above can be checked against what has arrived since.

Traffic to this site only, read from the live record on 18 August 2026. Paths are reproduced as received. A request for a credential file is not evidence that any file was exposed — none of the paths above returned one here.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Do fake AI crawlers scan for credential files?

Yes. Of 874 recorded requests claiming a named AI crawler, 69 asked for files that hold secrets such as .env, .ssh material or Terraform state.

How do you know those requests were not the real crawler?

The source addresses fell outside the ranges the named operators publish, so the identity claim fails against the operator's own feed.

Does this change how I should block crawlers?

It argues for verifying by address rather than name. A robots.txt rule is honoured by the real operator and ignored entirely by whoever is wearing its name.

Do these requests mean my .env file was exposed?

No. A request for a credential file is a probe, not proof of exposure. Check that the file is not served: request it yourself from outside your network and confirm you get a refusal or a not-found, never the file contents.

Why do scanners use AI crawler names?

Because some sites let named crawlers through rules that stop anonymous clients. A name like Google-Extended or GPTBot is a string anyone can type, so scanners borrow names a cautious site is least likely to block.

Is a 403 to a forged crawler a problem for SEO?

No. The real crawler comes from its operator's published address ranges and never fails the check, so refusing requests that fail it costs genuine crawlers nothing.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works