Guides · 2026-10-02 · By VSNARY | Emmanuel Orta · 3 views
Web Bot Auth: how signed bot requests work, and who signs
IP lists stop working when agents share cloud addresses. Web Bot Auth has the agent sign each request with a key it publishes. How a site verifies one, the CDN caching trap that breaks key rotation, and which operators document signing today.
Web Bot Auth is an IETF draft for authenticating bots with HTTP Message Signatures (RFC 9421). The agent signs each request with an Ed25519 key and sends Signature, Signature-Input and Signature-Agent, which names the domain whose /.well-known/http-message-signatures-directory holds the public keys. A site fetches that directory, matches the keyid, checks the time window and verifies the signature. As of 8 October 2026, OpenAI, Google (experimentally), Cloudflare and Cloudflare's first signed-agent cohort document signing; Anthropic and Perplexity document IP ranges instead.
Reverse DNS and published IP ranges both assume the operator owns its addresses. That fits crawlers, which run from their operator's own machines. It fits agents badly: an agent working for a person often runs in a shared cloud browser whose addresses also serve other customers. Web Bot Auth replaces the network check with a cryptographic one. The agent signs each request, names the domain that holds its keys, and the site checks the signature against those keys.
Who signs their requests today? #
Each row comes from the operator's own documentation or announcement, read on 8 October 2026. Operators were included only where their documentation states that they sign, with the identity they state.
| Operator | Agent | Signs as | Source |
|---|---|---|---|
| OpenAI | ChatGPT's cloud browser (ChatGPT Work) | https://chatgpt.com; keys at chatgpt.com/.well-known/http-message-signatures-directory | OpenAI Help Center, agent allowlisting |
| Google-Agent | Experimenting with https://agent.bot.goog; also publishes user-triggered-agents.json ranges | Google crawler documentation | |
| Cloudflare | Browser Rendering | Signed since Web Bot Auth launched | Cloudflare, “The age of agents”, 28 Aug 2025 |
| Block, Browserbase, Anchor Browser | Goose, and two remote-browser platforms | First cohort of Cloudflare's signed-agent category, with ChatGPT agent | Same announcement |
| CrawlCheck | Our own crawler | Signs its own requests; directory at /.well-known/http-message-signatures-directory | This site |
| Operator | Documents instead | Signing documented? |
|---|---|---|
| Anthropic | ClaudeBot, Claude-User and Claude-SearchBot by user agent; ranges at claude.com/crawling/bots.json | No |
| Perplexity | PerplexityBot and Perplexity-User by user agent; ranges at perplexity.com/perplexitybot.json and perplexity-user.json | No |
Anthropic's and Perplexity's crawler pages were read for any mention of Web Bot Auth, HTTP message signatures or Signature-Agent and contain none. That is not a weakness for their crawlers, which run from those ranges and verify cleanly. It matters for agents: user-triggered fetchers are the names forged most often, and as assistants move from fetching a page to operating a browser for a person, a range file stops describing where the traffic comes from. A signature is the only identity that travels with the request. The large search crawlers still rely on DNS and ranges, covered in how to verify crawlers by IP.
What does a signed request carry? #
The operator generates an Ed25519 key pair and publishes the public key in a JSON Web Key Set at /.well-known/http-message-signatures-directory on its own domain, served as application/http-message-signatures-directory+json. Each key is identified by its JWK thumbprint.
| Header | Carries | Example content |
|---|---|---|
Signature-Agent | where the key directory lives | "https://crawler.example" |
Signature-Input | which components were signed and the parameters | ("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";tag="web-bot-auth" |
Signature | the signature bytes, base64 | sig1=:…: |
created and expires bound the window, so a captured request cannot be replayed indefinitely. keyid names the key by thumbprint. tag="web-bot-auth" marks the purpose, so a signature made for something else cannot be reused here. An optional nonce allows replay detection inside the window.
How does a site verify a signed request? #
- Parse
Signature-Input; confirm the tag, the algorithm and that now is inside the created/expires window. - Read
Signature-Agent, fetch that origin's directory over HTTPS, and cache it briefly. - Find the key whose thumbprint matches
keyid. If none matches, refetch the directory once before rejecting. - Rebuild the signature base from the listed components exactly as RFC 9421 specifies and verify the Ed25519 signature.
- Allow or block on the identity in
Signature-Agent, never on the user-agent string next to it.
Cloudflare runs this check for bots registered with it and treats a verified signature as a verified bot. CrawlCheck's free tools include a Web Bot Auth checker that fetches an agent's directory and verifies a captured request's signature, window, tag and key.
The CDN caching trap #
Rotation works by listing old and new keys together: publish the new key, start signing with it, wait for verifiers' cached copies of the directory to expire, then withdraw the old key. That only works if the directory's cache lifetime is short, so verifiers should cache for minutes or hours, not days.
The trap is on the operator side. A CDN zone setting that overrides browser cache lifetimes can rewrite the directory's short max-age into a much longer one on cached responses. Verifiers then hold the old key set long after rotation and reject requests signed with the new key. Serve the directory with explicit cache headers, exempt its path from zone-wide cache overrides, and check the headers clients actually receive, not what the origin sends. The same edge behaviour affects other machine files; see stale cached machine files.
What does a valid signature prove? #
- It proves the request came from whoever holds the private key published in the named directory, within the signature's window.
- It does not prove what the agent will do with the page, whether it follows robots.txt, or that the person behind it is who they say. A signature from a domain you have never heard of verifies cleanly and tells you very little.
- It replaces the user-agent string as the thing to allow or block on, for operators that sign.
Should a site require signatures? #
Not yet. Most crawlers that matter for search and AI answers do not sign, so requiring a signature would block them. Verify signatures where present, treat a verified signature as a stronger signal than an IP match, and keep DNS and IP checks for everything else. The drafts are still changing, so pin your implementation to a draft version. If you run a crawler yourself, signing costs one key pair, one static directory and a few lines around each request.
CrawlCheck’s dataset when this page loaded: 582,334 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is Web Bot Auth?
An IETF draft that lets automated clients sign HTTP requests with HTTP Message Signatures (RFC 9421), so a site can verify which operator sent a request without relying on IP addresses.
Which AI agents use Web Bot Auth?
As of 8 October 2026: ChatGPT's cloud browser (signs as https://chatgpt.com), Google-Agent (Google is experimenting with https://agent.bot.goog), Cloudflare Browser Rendering, and the agents in Cloudflare's first signed-agent cohort: ChatGPT agent, Block's Goose, Browserbase and Anchor Browser.
Does Claude or Perplexity sign its requests?
Not according to their documentation as read on 8 October 2026. Anthropic and Perplexity identify their crawlers and fetchers by user agent and publish IP ranges instead.
Which headers does a Web Bot Auth request carry?
Signature, Signature-Input and Signature-Agent. Signature-Input lists the signed components and the created, expires, keyid, alg and tag parameters; Signature-Agent points to the key directory.
Why do signed requests fail after a key rotation?
Usually because a verifier, or a CDN in front of the directory, cached the old key set too long. Keep the directory's cache lifetime short, stop zone-wide cache overrides from lengthening it, and refetch the directory once when a keyid is not found.
Does a valid signature mean the agent is trustworthy?
No. It proves the request was signed with a key published at a given origin. Whether that operator behaves well is a separate question of reputation and policy.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.