Agent view · Guides · 2026-10-02
Web Bot Auth: how signed bot requests work, and who signs
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/web-bot-auth-how-signed-crawler-requests-work: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/web-bot-auth-how-signed-crawler-requests-work>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 16,483 |
| words | 2,257 |
| sha-256 | 4211a930ab8d47268625b012d05919e3c9ba2b5aa0e1f6a25ab5f81a05d9f3bc |
| repr-digest | sha-256=:QhGpMKuNRyaGJbAS0FkZ48m6K1qg4faiWrX4GgXZ87w=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311479;expires=1795315079;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="s1q6ZV9A6YxrscbSvYFbuQ";tag="crawlcheck-page" |
| signature | sig1=:Kcil+eHJriE753/uM8Mylu95DYdckjlB6QMiOpjp/F9WgOvREPZzR+ud+DqyHopS73V2tv36WRztLcNQsSPwCQ==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-02 · By [VSNARY | Emmanuel Orta](/author) · 3 views
# Web Bot Auth: how signed bot requests work, and who signs
IP lists stop working when agents share cloud addresses. Web Bot Auth has the agent sign each request with a key it publishes. How a site verifies one, the CDN caching trap that breaks key rotation, and which operators document signing today.
Web Bot Auth is an IETF draft for authenticating bots with HTTP Message Signatures (RFC 9421). The agent signs each request with an Ed25519 key and sends Signature, Signature-Input and Signature-Agent, which names the domain whose /.well-known/http-message-signatures-directory holds the public keys. A site fetches that directory, matches the keyid, checks the time window and verifies the signature. As of 8 October 2026, OpenAI, Google (experimentally), Cloudflare and Cloudflare's first signed-agent cohort document signing; Anthropic and Perplexity document IP ranges instead.
Check your own site free
Check my site
About a minute. First scan free with your email, nothing installed.
On this page
- Who signs their requests today?
- What does a signed request carry?
- How does a site verify a signed request?
- The CDN caching trap
- What does a valid signature prove?
- Should a site require signatures?
Reverse DNS and published IP ranges both assume the operator owns its addresses. That fits crawlers, which run from their operator's own machines. It fits agents badly: an agent working for a person often runs in a shared cloud browser whose addresses also serve other customers. [Web Bot Auth](/glossary/web-bot-auth) replaces the network check with a cryptographic one. The agent signs each request, names the domain that holds its keys, and the site checks the signature against those keys.
## Who signs their requests today? #
Each row comes from the operator's own documentation or announcement, read on 8 October 2026. Operators were included only where their documentation states that they sign, with the identity they state.
| Operator | Agent | Signs as | Source |
| --- | --- | --- | --- |
| OpenAI | ChatGPT's cloud browser (ChatGPT Work) | `https://chatgpt.com`; keys at `chatgpt.com/.well-known/http-message-signatures-directory` | OpenAI Help Center, agent allowlisting |
| Google | [Google-Agent](/glossary/google-agent) | Experimenting with `https://agent.bot.goog`; also publishes `user-triggered-agents.json` ranges | Google crawler documentation |
| Cloudflare | Browser Rendering | Signed since Web Bot Auth launched | Cloudflare, “The age of agents”, 28 Aug 2025 |
| Block, Browserbase, Anchor Browser | Goose, and two remote-browser platforms | First cohort of Cloudflare's signed-agent category, with ChatGPT agent | Same announcement |
| CrawlCheck | Our own crawler | Signs its own requests; directory at [/.well-known/http-message-signatures-directory](/.well-known/http-message-signatures-directory) | This site |
| Operator | Documents instead | Signing documented? |
| --- | --- | --- |
| Anthropic | ClaudeBot, Claude-User and Claude-SearchBot by [user agent](/glossary/user-agent); ranges at `claude.com/crawling/bots.json` | No |
| Perplexity | PerplexityBot and Perplexity-User by user agent; ranges at `perplexity.com/perplexitybot.json` and `perplexity-user.json` | No |
Anthropic's and Perplexity's crawler pages were read for any mention of Web Bot Auth, HTTP message signatures or Signature-Agent and contain none. That is not a weakness for their crawlers, which run from those ranges and verify cleanly. It matters for agents: user-triggered fetchers are the names forged most often, and as assistants move from fetching a page to operating a browser for a person, a range file stops describing where the traffic comes from. A signature is the only identity that travels with the request. The large search crawlers still rely on DNS and ranges, covered in [how to verify crawlers by IP](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers).
## What does a signed request carry? #
The operator generates an Ed25519 key pair and publishes the public key in a JSON Web Key Set at `/.well-known/http-message-signatures-directory` on its own domain, served as `application/http-message-signatures-directory+json`. Each key is identified by its JWK thumbprint.
| Header | Carries | Example content |
| --- | --- | --- |
| `Signature-Agent` | where the key directory lives | `"https://crawler.example"` |
| `Signature-Input` | which components were signed and the parameters | `("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";tag="web-bot-auth"` |
| `Signature` | the signature bytes, base64 | `sig1=:…:` |
`created` and `expires` bound the window, so a captured request cannot be replayed indefinitely. `keyid` names the key by thumbprint. `tag="web-bot-auth"` marks the purpose, so a signature made for something else cannot be reused here. An optional `nonce` allows replay detection inside the window.
## How does a site verify a signed request? #
- Parse `Signature-Input`; confirm the tag, the algorithm and that now is inside the created/expires window.
- Read `Signature-Agent`, fetch that origin's directory over HTTPS, and cache it briefly.
- Find the key whose thumbprint matches `keyid`. If none matches, refetch the directory once before rejecting.
- Rebuild the signature base from the listed components exactly as RFC 9421 specifies and verify the Ed25519 signature.
- Allow or block on the identity in `Signature-Agent`, never on the user-agent string next to it.
Cloudflare runs this check for bots registered with it and treats a verified signature as a verified bot. CrawlCheck's free tools include a Web Bot Auth checker that fetches an agent's directory and verifies a captured request's signature, window, tag and key.
## The CDN caching trap #
Rotation works by listing old and new keys together: publish the new key, start signing with it, wait for verifiers' cached copies of the directory to expire, then withdraw the old key. That only works if the directory's cache lifetime is short, so verifiers should cache for minutes or hours, not days.
The trap is on the operator side. A [CDN](/glossary/cdn) zone setting that overrides browser cache lifetimes can rewrite the directory's short `max-age` into a much longer one on cached responses. Verifiers then hold the old key set long after rotation and reject requests signed with the new key. Serve the directory with explicit cache headers, exempt its path from zone-wide cache overrides, and check the headers clients actually receive, not what the origin sends. The same edge behaviour affects other machine files; see [stale cached machine files](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge).
## What does a valid signature prove? #
- **It proves** the request came from whoever holds the private key published in the named directory, within the signature's window.
- **It does not prove** what the agent will do with the page, whether it follows [robots.txt](/glossary/robots-txt), or that the person behind it is who they say. A signature from a domain you have never heard of verifies cleanly and tells you very little.
- **It replaces** the user-agent string as the thing to allow or block on, for operators that sign.
## Should a site require signatures? #
Not yet. Most crawlers that matter for search and AI answers do not sign, so requiring a signature would block them. Verify signatures where present, treat a verified signature as a stronger signal than an IP match, and keep DNS and IP checks for everything else. The drafts are still changing, so pin your implementation to a draft version. If you run a crawler yourself, signing costs one key pair, one static directory and a few lines around each request.
CrawlCheck’s dataset when this page loaded: 582,334 sites measured.
Written by
[Emmanuel Orta](/author) · VSNARY
I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.
[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)
Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fweb-bot-auth-how-signed-crawler-requests-work%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fweb-bot-auth-how-signed-crawler-requests-work%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Web%20Bot%20Auth%3A%20how%20signed%20bot%20requests%20work%2C%20and%20who%20signs)[Email](mailto:?subject=Web%20Bot%20Auth%3A%20how%20signed%20bot%20requests%20work%2C%20and%20who%20signs&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fweb-bot-auth-how-signed-crawler-requests-work)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)
## Questions this post answers
### What is Web Bot Auth?
An IETF draft that lets automated clients sign HTTP requests with HTTP Message Signatures (RFC 9421), so a site can verify which operator sent a request without relying on IP addresses.
### Which AI agents use Web Bot Auth?
As of 8 October 2026: ChatGPT's cloud browser (signs as https://chatgpt.com), Google-Agent (Google is experimenting with https://agent.bot.goog), Cloudflare Browser Rendering, and the agents in Cloudflare's first signed-agent cohort: ChatGPT agent, Block's Goose, Browserbase and Anchor Browser.
### Does Claude or Perplexity sign its requests?
Not according to their documentation as read on 8 October 2026. Anthropic and Perplexity identify their crawlers and fetchers by user agent and publish IP ranges instead.
### Which headers does a Web Bot Auth request carry?
Signature, Signature-Input and Signature-Agent. Signature-Input lists the signed components and the created, expires, keyid, alg and tag parameters; Signature-Agent points to the key directory.
### Why do signed requests fail after a key rotation?
Usually because a verifier, or a CDN in front of the directory, cached the old key set too long. Keep the directory's cache lifetime short, stop zone-wide cache overrides from lengthening it, and refetch the directory once when a keyid is not found.
### Does a valid signature mean the agent is trustworthy?
No. It proves the request was signed with a key published at a given origin. Whether that operator behaves well is a separate question of reputation and policy.
## Related findings
[2026-10-15 Share of preflight, October 2026: 0 of 5,643 requests Seven connected sites verified 5,643 AI agent requests in October 2026 to date. None carried a CrawlCheck receipt; the 68 accepted](/blog/share-of-preflight-october-2026)[2026-10-07 Eight weeks of AI crawlers on one site: who was really who 24,627 verified crawler requests to crawlcheck.io in eight weeks: who was real, which names only came forged, and what verified ag](/blog/eight-weeks-of-agent-conduct-who-was-really-who)[2026-10-18 How to verify Amazonbot and Amzn-SearchBot by IP address Amazon publishes Amazonbot and Amzn-SearchBot addresses on web pages. How to check a request against them, and how many claims we](/guides/how-to-verify-amazonbot-and-amzn-searchbot)
The main product
Found this on your own site? **We fix it for $100.**
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)
Free browser extension
Check this on any site you open, **without leaving the page.**
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)
Write for the library
Measured something yourself? **Submit an article.**
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week
Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).
## Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.
Send for review
Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.
[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)
We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)
#### How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).
**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.
**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.
**Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [Removal](/removal) | **Registry:** [woorank.com](/registry/woorank.com) · [agent-exec.448c.org](/registry/capabilities/agent-exec.448c.org) | **Datasets:** [West Virginia](/datasets/state/west-virginia) · [Technologies: Wp rocket](/datasets/technologies/wp-rocket) | **Guides:** [Localbusiness schema for service area businesses](/guides/localbusiness-schema-for-service-area-businesses) | **Glossary:** [AI classifiers: Answer justification](/glossary/ai-classifiers/answer-justification) · [OCR](/glossary/ocr)
[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/web-bot-auth-how-signed-crawler-requests-work.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.