Agent view · Findings · 2026-10-04
402 Payment Required: what top sites send AI crawlers
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 17,727 |
| words | 2,605 |
| sha-256 | 3a0f2a60495c0b53c06c5b491a8144b449de47415cc751f4af611d621ba61d15 |
| repr-digest | sha-256=:Og8qYElcC1PAbFtJGoFEtEneR0Fcx1H0r2EdYhumHRU=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311497;expires=1795315097;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="xpDXOAqObDNJ8Y_5-sixeg";tag="crawlcheck-page" |
| signature | sig1=:lV5C60WaJrzTX9EW7dbavAytLCiTVMOvhPiHdKVVH6y4t2HClOyI6YzLcLQIFR9Nvhp1a7YiSF2T1JnvHah7Ag==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Exclusive finding · Findings · 2026-10-04 · By [VSNARY | Emmanuel Orta](/author) # 402 Payment Required: what top sites send AI crawlers Three of the top 1,000 sites answer AI crawlers with HTTP 402. None quotes a price: one is a refusal, one a toll, one contradicts its own robots.txt. On 4 October 2026, three operators in the Tranco top 1,000 answered AI crawlers with HTTP 402 Payment Required: x.com, forbes.com and independent.co.uk. None quoted a price. On x.com the 402 is a refusal under another code; on Forbes it is a toll through a separate paid-access host; on The Independent it contradicts robots.txt, which allows the crawlers it charges. To a crawler, an unpriced 402 works like a block. CrawlCheck exclusive finding Sample Tranco top 1,000, four crawler names, every response code read Measured 4 October 2026 What it established The first census of HTTP 402 sent to AI crawlers: three operators use it, none quotes a price, and the three mean three different things. An unpriced 402 is a block. One of [23 findings measured only here](/blog?kind=exclusive). Every figure comes from CrawlCheck’s own readings, dated, with the method at the foot. Have your own? [Submit a finding](/submit-finding). On this page - What did the census find? - What does a 402 mean on each site? - What is a priced 402 supposed to look like? - Why does this matter if you run a site? - How do I check my own site? - What does this not show? Three top sites that answered 402, and what the 402 contained x.com (and twitter.com), rank 48 (15) Answered 402 GPTBot, OAI-SearchBot, PerplexityBot. ClaudeBot got 403 robots.txt for those crawlers disallows all four What the 402 contained 55 bytes of JSON: “Please contact the site owner for access.” forbes.com, rank 222 Answered 402 GPTBot, ClaudeBot, OAI-SearchBot, after a redirect to tollbit.forbes.com robots.txt for those crawlers disallows GPTBot and ClaudeBot, allows OAI-SearchBot What the 402 contained JSON: no access “without a valid TollBit Token”, with a link to tollbit.dev independent.co.uk, rank 727 Answered 402 ClaudeBot, PerplexityBot robots.txt for those crawlers allows all four What the 402 contained An HTML page titled “Payment Required” HTTP 402 Payment Required sat unused in the status-code list for decades. It is now how some large sites answer AI crawlers. We read the top 1,000 sites of the Tranco list on 4 October 2026, each homepage once as a browser and once as each of four AI crawlers: GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot. Three operators answered at least one of those crawlers with 402. None of the three quoted a price. **Disclosure:** CrawlCheck publishes this and runs the census it comes from. Every reply described here was read by our census and then fetched again by hand on 4 October 2026. A crawler name in a request is a claim, not an identity: these are the answers a server gave to that claim from our address, which may differ from what the real crawler receives from its own published addresses. ## What did the census find? # Of the 991 domains read, 524 served a homepage to a plain browser request. Most of the rest are infrastructure domains with no website at all, such as [CDN](/glossary/cdn), DNS and update hosts; others refused the read or timed out. Among the 524, four domains answered an AI crawler with 402. Two of them are one service: twitter.com redirects to x.com. | Site | Rank | Answered 402 | [robots.txt](/glossary/robots-txt) for those crawlers | What the 402 contained | | --- | --- | --- | --- | --- | | x.com (and twitter.com) | 48 (15) | GPTBot, OAI-SearchBot, PerplexityBot. ClaudeBot got 403 | disallows all four | 55 bytes of JSON: “Please contact the site owner for access.” | | forbes.com | 222 | GPTBot, ClaudeBot, OAI-SearchBot, after a redirect to tollbit.forbes.com | disallows GPTBot and ClaudeBot, allows OAI-SearchBot | JSON: no access “without a valid TollBit [Token](/glossary/token)”, with a link to tollbit.dev | | independent.co.uk | 727 | ClaudeBot, PerplexityBot | allows all four | An HTML page titled “Payment Required” | The x.com and Independent answers were the same in the two earlier runs. Forbes did not answer the census at all in those runs, so its pattern is one day old in our data, and we confirmed it by hand. ## What does a 402 mean on each site? # The same three digits mean three different things here. On x.com, 402 is a [refusal](/glossary/refusal) under another name: robots.txt already disallows the crawler, there is no price and no way to pay, and the message says to contact the site owner. A crawler can do nothing with it except stop. On forbes.com, 402 is a toll. The crawler is redirected to a separate host run for a paid-access service, and that host refuses it without a token from the service. There is a path to payment, but it runs through an account with that service, not through anything in the reply itself. On independent.co.uk, 402 contradicts robots.txt. The file invites ClaudeBot and PerplexityBot, and the server asks them for payment. GPTBot and OAI-SearchBot, also invited, got the full homepage. A site owner reading only robots.txt would believe all four crawlers can read the site. The same robots-says-yes, server-says-no pattern at larger scale is in [robots.txt says yes, the server says no](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026). ## What is a priced 402 supposed to look like? # There is a published design for 402 as a real price. Cloudflare's pay per crawl, in closed beta, answers an AI crawler with 402 and a `crawler-price` header. The crawler retries with `crawler-exact-price` or `crawler-max-price`, signs the request with [Web Bot Auth](/glossary/web-bot-auth) so the payment can be tied to a verified operator, and on success gets 200 with a `crawler-charged` header naming what was billed. A failed attempt comes back as 402 with a `crawler-error` header. The signing step is explained in [Web Bot Auth explained](/guides/web-bot-auth-how-signed-crawler-requests-work). None of the four replies in the top 1,000 carried a `crawler-price` header. So on 4 October 2026, every 402 an AI crawler got from a top-1,000 homepage was either a refusal or a toll run through another service. None was a price quoted in the reply. ## Why does this matter if you run a site? # Crawlers do not treat 402 as an invitation. To a search or answer crawler, a 402 on your homepage is a page it could not read, like a 403. If your goal is to be cited, a 402 does the same job as a block, whatever the intent behind it. If a 402 comes from a setting rather than a decision, it is easy to miss. robots.txt says yes, a browser sees the page, and only the crawler sees the bill. The Independent's split, two crawlers served and two charged, is the kind of result that usually comes from a rule set per crawler at the edge or in a paid-access integration, not from anything in robots.txt. If you mean to charge, the reply should say so in a way a crawler can act on: a price, and a way to pay that a crawler operator can use. A bare 402 with a contact address tells an automated client nothing it can do. ## How do I check my own site? # Request your homepage with a crawler's user-agent and look at the status line, any redirect and the headers: ``` curl -sI -L -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://www.example.com/ ``` Repeat with each crawler you care about and once with a normal browser user-agent. Then compare: | What you see | What it means | | --- | --- | | 200 for the browser and for the crawler | The crawler is served. robots.txt still decides whether a polite crawler reads on. | | 402 with a `crawler-price` header | A priced offer the crawler can accept or decline. | | 402 with no price, or a redirect to another host that answers 402 | A refusal, or a toll run by a third-party service. Find which setting or integration produces it. | | 402 or 403 for a crawler your robots.txt allows | The two disagree. Decide which one states your intent and change the other. | One caution. Some servers check that a request claiming to be a crawler comes from that crawler's published addresses, and refuse it from anywhere else. If your server refuses a request claiming to be Googlebot as well, your test is measuring that address check, not your crawler policy. Test from a normal connection, and read your access logs for what the real crawler got. A [CrawlCheck scan](/) makes this comparison for every crawler at once, and the guides on [whether Cloudflare is blocking AI crawlers](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site) and [how to block AI crawlers on purpose](/guides/how-to-block-ai-crawlers) cover the settings that usually sit behind a mismatch. ## What does this not show? # This is one homepage per site, read from our address, with four crawler names. It does not show what the real crawlers receive from their own addresses, what any of these sites charge or to whom, or whether any crawler has paid. It shows only what the status line and the reply said when we asked. The census figures for the whole top 1,000 are on [the data page](/data#census). CrawlCheck’s dataset when this page loaded: 582,334 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=402%20Payment%20Required%3A%20what%20top%20sites%20send%20AI%20crawlers)[Email](mailto:?subject=402%20Payment%20Required%3A%20what%20top%20sites%20send%20AI%20crawlers&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### What does HTTP 402 Payment Required mean for a crawler? It means the server wants payment before it serves the page. A crawler that cannot pay treats it like any other refusal: the page is not read, so it cannot be indexed or cited. ### Do sites actually send 402 to AI crawlers? Yes. On 4 October 2026, three operators in the Tranco top 1,000 answered at least one AI crawler with 402: x.com, forbes.com and independent.co.uk. None of their replies quoted a price. ### What does a priced 402 look like? Cloudflare's pay per crawl, in closed beta, answers with 402 and a crawler-price header. The crawler retries with crawler-exact-price or crawler-max-price, signed with Web Bot Auth, and gets 200 with a crawler-charged header when payment succeeds. ### Can robots.txt allow a crawler that the server charges? Yes, and it happened in the top 1,000: The Independent's robots.txt allows ClaudeBot and PerplexityBot, and its server answers both with 402. robots.txt and the server are separate settings and can disagree. ### How do I check what my site sends an AI crawler? Request your homepage with curl and the crawler's user-agent, and compare the status, redirect and headers with a browser request. If a request claiming to be Googlebot is refused too, you are testing an address check, not your crawler policy. ## Related findings [2026-10-22 One 503 is not a blocked site: refusal or hiccup? Why one 503 on robots.txt or a sitemap is not proof a site blocks crawlers, and a three-request test that tells a refusal from a h](/blog/one-503-is-not-a-blocked-site-how-to-tell-a-refusal-from-a-hiccup)[2026-08-28 Do AI crawlers and agents respect robots.txt? Most named AI crawlers obey robots.txt; one verified crawler broke ours. Why sites lose to unreadable files and edge refusals, and](/guides/do-ai-crawlers-respect-robots-txt)[2026-11-03 One Cloudflare setting is behind most of the new AI training blocks on top sites Census of the Tranco top 10,000, October to November 2026: which AI crawlers sites newly blocked, how much of it came from Cloudfl](/blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [arti-box.fr](/registry/arti-box.fr) · [dayoneapp.com](/registry/dayoneapp.com) · [heckenlau-immobilien.de](/registry/heckenlau-immobilien.de) · [minisplitreef.com](/registry/minisplitreef.com) · [theagenttimes.com](/registry/theagenttimes.com) · [Place: Austin](/registry/place/tx/austin) | **Glossary:** [verdict over DNS](/glossary/verdict-over-dns) · [Permissions-Policy](/glossary/permissions-policy) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/ai-crawlers-get-402-payment-required-from-top-sites-none-quote-a-price.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.