Findings · 2026-10-08 · By VSNARY | Emmanuel Orta · 0 views

AI agents acting for people: 613 verified visits, one read robots.txt

The fetchers an assistant sends when a person asks about a page, counted on one site and checked against each operator’s published address ranges. They are a small share of AI traffic, the most forged names in it, and they handle robots.txt exactly as their operators document.

When a person asks an assistant about a page, the assistant sends a fetcher to read it: ChatGPT-User for OpenAI, Claude-User for Anthropic, Perplexity-User for Perplexity, Google-Agent for agents hosted by Google. These are the first agents of the agentic web that most sites ever see. This post counts them on one site, crawlcheck.io, from the public conduct record, read on 8 October 2026 at 8:50 AM MDT.

A small share of AI traffic #

The three fetchers made 613 verified requests. Every verified named agent together made 25,718. User-triggered fetches are 2.4% of the verified total; the rest is crawlers indexing and training on their own schedules: Googlebot 5,596, PerplexityBot 5,322, GPTBot 4,016, ClaudeBot 3,748. The agentic web is arriving one question at a time, and on this site ChatGPT-User asks most of them.

The most forged names in the log #

For index crawlers, between one and five claims in a hundred are forged (the full table). For user-triggered fetchers the share climbs as the real traffic falls: 12% of ChatGPT-User claims, 40% of Claude-User claims and 86% of Perplexity-User claims came from addresses outside the operator’s published ranges. There were six times more fake Perplexity-User requests here than real ones.

The reason is policy. Sites that block crawlers often let a person’s assistant through, so the assistant’s name is the one worth wearing. A rule that trusts ChatGPT-User by name trusts everyone who has read the same documentation.

robots.txt: each behaved as its operator documents #

Claude-User fetched robots.txt 8 times in eight weeks. ChatGPT-User and Perplexity-User fetched it zero times across 534 verified requests between them. That matches what each operator publishes: Anthropic says Claude-User honours robots.txt; OpenAI says robots.txt rules “may not apply” to ChatGPT-User because a user started the action; Perplexity says Perplexity-User “generally ignores” them.

Not reading the file did not mean breaking it here. No verified request from any of the three fetched a path this site disallows. That is one site and three disallowed paths, so it is a baseline, not a verdict on anyone else’s rules. What the documentation says is the stronger guide to what they will do on yours.

Who did not arrive #

Google-Agent, the name Google gives agents hosted on its infrastructure, did not make a single request here in eight weeks, verified or forged. MistralAI-User was named 9 times; none of the nine could be tied to an address range this record reads, so none is judged. Absence on one site is not absence everywhere; it is what the agentic web looked like from Denver this autumn.

Other sites are already checking #

Sites can ask CrawlCheck whether an address and user agent belong to the operator they claim, through Visitor Resolve. In the last 14 days other sites checked ChatGPT-User 51 times: 48 were verified and 3 spoofed. Claude-User was checked 3 times (1 verified) and Perplexity-User twice (both spoofed).

What to do with this #

• Never grant a user-triggered name more access than a crawler without checking the address. Each of the three operators publishes ranges: openai.com/chatgpt-user.json, claude.com/crawling/bots.json, perplexity.com/perplexity-user.json.
• If a page must not be fetched by an assistant acting for someone, enforce it at the edge; robots.txt binds one of the three.
• Expect the share to grow and the forgery rate to stay high, because the name is valuable.

Method #

Every request to crawlcheck.io whose user agent names a known agent is checked on arrival against the address ranges its operator publishes. Verified means inside the ranges; forged means the name from outside every range the operator publishes for it. Forged counts here exclude requests to report and tool paths, which are people checking our own output. The record is signed and verifies offline; the figures above are its values at 14:50 UTC on 8 October 2026.

Questions this post answers

Does ChatGPT-User respect robots.txt?

OpenAI’s documentation says robots.txt rules may not apply to ChatGPT-User because a user initiated the request. On crawlcheck.io, ChatGPT-User made 526 verified requests in eight weeks without fetching robots.txt once, and none of them fetched a disallowed path.

How often is Perplexity-User spoofed?

On crawlcheck.io between 12 August and 8 October 2026, 50 of 58 requests naming Perplexity-User (86.2%) came from outside the address ranges Perplexity publishes. Only 8 were the real fetcher.

How do I verify a ChatGPT-User or Claude-User request?

Check the source address against the range file the operator publishes: openai.com/chatgpt-user.json for ChatGPT-User, claude.com/crawling/bots.json for Anthropic’s agents and perplexity.com/perplexity-user.json for Perplexity-User. The user agent string alone proves nothing.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works