Web Bot Auth
A method for bots to cryptographically sign HTTP requests so recipients can authenticate identity without relying only on network location.
What Web Bot Auth does not show
A valid signature proves key control, not benign purpose.
Where Web Bot Auth comes up on this site
- Web Bot Auth: how signed bot requests work, and who signsWeb Bot Auth replaces the network check with a cryptographic one.
- AI crawler list: what each bot does and how to verify itSigned requests are covered in Web Bot Auth explained .
- How to block AI crawlers like GPTBot and ClaudeBotAgents increasingly sign their requests under Web Bot Auth , and a signature is the identity to block or allow on.
- Verify GPTBot, ClaudeBot and PerplexityBot: tools comparedCloudflare verifies every request at the edge as it arrives and can act on it, including through Web Bot Auth signatures.
- How an AI agent should verify a website before it actsWhen the domain sends its own agents, signed requests under Web Bot Auth (built on RFC 9421 HTTP Message Signatures ) identify them better than a user agent string anyone can type.
Definitions that name Web Bot Auth
HTTP Message Signatures · operator origin · signed agent
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Sitemap directive · Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range
Questions about Web Bot Auth
What is Web Bot Auth?
A method for bots to cryptographically sign HTTP requests so recipients can authenticate identity without relying only on network location.
What does Web Bot Auth not show or guarantee?
A method for bots to cryptographically sign HTTP requests so recipients can authenticate identity without relying only on network location. A valid signature proves key control, not benign purpose.
Which area of the glossary does Web Bot Auth belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.