Agent conduct

# What each agent actually did, from requests it really sent.

Every request to this site that names a tracked agent is checked against the address ranges its operator publishes. Only verified requests count toward an agent’s record; a request using its name from anywhere else is counted against the impersonator. One property’s census, signed at [/api/v1/conduct](/api/v1/conduct).

194,897

verified requests

1,524

forged claims, counted against the forger

14

agents with a clean robots.txt record

6

agents that fetched a disallowed path

For sites and agents

## How the record is used

1 · A site

### Is this visitor really it?

Visitor Resolve answers for one request, from the same published ranges: verified, spoofed or unverifiable.

[Visitor Resolve](/glossary/visitor-resolve)[/api/v1/visitor](/docs/api#api-v-visitor)

2 · An operator

### How your crawler is seen

Each agent has its own signed record: verified requests, forged claims, robots.txt and its busiest minute here.

[The full record](/api/v1/conduct)[Crawler catalogue](/ai-crawlers)

3 · Everyone

### Across many sites

The monthly Agent Traffic Index counts which agents reached which reporting sites, from the same kind of verification.

[Traffic index](/api/v1/traffic-index)[All open data](/data)

Right now

## Active today

Verified requests per UTC day, counted from 18 Oct 2026.

Agent of the day (2026-11-21): **Amazonbot** (Amazon), 2,227 verified requests.

| # | Agent | Today (2026-11-22) | Yesterday | Grade |
| --- | --- | --- | --- | --- |
| 1 | [AhrefsBot](/agents/conduct/ahrefsbot)
Ahrefs | 162 | 147 | D |
| 2 | [YandexBot](/agents/conduct/yandexbot)
Yandex | 23 | 485 | A |
| 3 | [PerplexityBot](/agents/conduct/perplexitybot)
Perplexity | 23 | 441 | D |
| 4 | [OAI-SearchBot](/agents/conduct/oai-searchbot)
OpenAI | 20 | 140 | A |
| 5 | [GPTBot](/agents/conduct/gptbot)
OpenAI | 20 | 63 | B |
| 6 | [Bingbot](/agents/conduct/bingbot)
Microsoft | 19 | 228 | D |
| 7 | [Applebot](/agents/conduct/applebot)
Apple | 14 | 90 | D |
| 8 | [Googlebot](/agents/conduct/googlebot)
Google | 10 | 110 | A |
| 9 | [Amazonbot](/agents/conduct/amazonbot)
Amazon | 4 | 2,227 | D |
| 10 | [DuckDuckBot](/agents/conduct/duckduckbot)
DuckDuckGo | 2 | 4 | A |
| 11 | [ChatGPT-User](/agents/conduct/chatgpt-user)
OpenAI | 1 | 10 | A |
| 12 | [Baiduspider](/agents/conduct/baiduspider)
Baidu | 1 | 0 | B |
| 13 | [Claude-User](/agents/conduct/claude-user)
Anthropic | 0 | 21 | A |
| 14 | [SeznamBot](/agents/conduct/seznambot)
Seznam | 0 | 20 | A |
| 15 | [GoogleOther](/agents/conduct/googleother)
Google | 0 | 10 | A |

Verified agents

## The record, agent by agent

A verdict needs 50 verified requests. The only rule judged is robots.txt; the busiest minute is shown, not judged, because this site publishes no crawl rate.

| Grade | Agent | Verified requests | Forged claims | robots.txt | Busiest minute | Conduct |
| --- | --- | --- | --- | --- | --- | --- |
| D | [Amazonbot](/agents/conduct/amazonbot)
Amazon · training | 37,134 | 119 | disallowed paths fetched | 34/min | violations observed |
| D | [PerplexityBot](/agents/conduct/perplexitybot)
Perplexity · indexing | 35,925 | 101 | disallowed paths fetched | 120/min | violations observed |
| A | [YandexBot](/agents/conduct/yandexbot)
Yandex · indexing | 29,922 | 20 | respected | 26/min | no violations observed |
| D | [AhrefsBot](/agents/conduct/ahrefsbot)
Ahrefs · tool | 15,070 | 3 | disallowed paths fetched | 64/min | violations observed |
| D | [ClaudeBot](/agents/conduct/claudebot)
Anthropic · training | 13,375 | 124 | disallowed paths fetched | 69/min | violations observed |
| D | [Applebot](/agents/conduct/applebot)
Apple · indexing | 12,683 | 84 | disallowed paths fetched | 8/min | violations observed |
| A | [Googlebot](/agents/conduct/googlebot)
Google · indexing | 12,437 | 266 | respected | 13/min | no violations observed |
| D | [Bingbot](/agents/conduct/bingbot)
Microsoft · indexing | 11,274 | 52 | disallowed paths fetched | 12/min | violations observed |
| B | [GPTBot](/agents/conduct/gptbot)
OpenAI · training | 10,547 | 184 | robots txt not fetched | 120/min | no violations observed |
| A | [OAI-SearchBot](/agents/conduct/oai-searchbot)
OpenAI · indexing | 9,105 | 111 | respected | 119/min | no violations observed |
| A | [ChatGPT-User](/agents/conduct/chatgpt-user)
OpenAI · live fetch | 1,778 | 90 | not applicable | 13/min | no violations observed |
| A | [Claude-User](/agents/conduct/claude-user)
Anthropic · live fetch | 1,607 | 55 | not applicable | 7/min | no violations observed |
| A | [SeznamBot](/agents/conduct/seznambot)
Seznam · indexing | 1,156 | 0 | respected | 3/min | no violations observed |
| B | [Baiduspider](/agents/conduct/baiduspider)
Baidu · indexing | 691 | 45 | robots txt not fetched | 5/min | no violations observed |
| A | [CCBot](/agents/conduct/ccbot)
Common Crawl · training | 636 | 9 | respected | 7/min | no violations observed |
| A | [Google-CWS](/agents/conduct/google-cws)
Google · store listing fetch | 478 | 0 | not applicable | 3/min | no violations observed |
| A | [Google-InspectionTool](/agents/conduct/google-inspectiontool)
Google | 361 | 5 | respected | 7/min | no violations observed |
| A | [Google-Agent](/agents/conduct/google-agent)
Google · live fetch | 281 | 0 | not applicable | 17/min | no violations observed |
| A | [DuckDuckBot](/agents/conduct/duckduckbot)
DuckDuckGo · indexing | 203 | 0 | respected | 2/min | no violations observed |
| A | [GoogleOther](/agents/conduct/googleother)
Google · training | 193 | 134 | respected | 5/min | no violations observed |
| — | [Perplexity-User](/agents/conduct/perplexity-user)
Perplexity · live fetch | 9 | 52 | insufficient data | 1/min | insufficient data |
| — | [Google-NotebookLM](/agents/conduct/google-notebooklm)
Google | 8 | 0 | insufficient data | 4/min | insufficient data |
| — | [Google-Read-Aloud](/agents/conduct/google-read-aloud)
 | 7 | 0 | insufficient data | 3/min | insufficient data |
| — | [Yeti](/agents/conduct/yeti)
Naver · indexing | 6 | 0 | insufficient data | 1/min | insufficient data |
| — | [AdsBot-Google](/agents/conduct/adsbot-google)
Google · tool | 5 | 0 | insufficient data | 2/min | insufficient data |
| — | [Google-Safety](/agents/conduct/google-safety)
 | 4 | 0 | insufficient data | 2/min | insufficient data |
| — | [Google-Site-Verification](/agents/conduct/google-site-verification)
Google · ownership check | 2 | 0 | insufficient data | not yet | insufficient data |
| — | [Claude-SearchBot](/agents/conduct/claude-searchbot)
Anthropic · indexing | 0 | 58 | insufficient data | not yet | insufficient data |
| — | [MistralAI-User](/agents/conduct/mistralai-user)
Mistral · live fetch | 0 | 0 | insufficient data | not yet | insufficient data |
| — | [FeedFetcher-Google](/agents/conduct/feedfetcher-google)
Google · feed fetch | 0 | 1 | insufficient data | not yet | insufficient data |
| — | [Google-CloudVertexBot](/agents/conduct/google-cloudvertexbot)
Google · tool | 0 | 3 | insufficient data | not yet | insufficient data |
| — | [Storebot-Google](/agents/conduct/storebot-google)
Google · indexing | 0 | 8 | insufficient data | not yet | insufficient data |

Verified = source address inside the operator’s published ranges. Forged claims = requests that used the agent’s name from outside them; they are counted against the impersonator, never the agent. Disallowed paths are /r/, /backup/, /admin-old/. Busiest minute = all of an operator’s addresses combined; rate and revalidation counters started 7 October 2026. Generated 2026-11-22 01:06 UTC.

Not assessed

## Seen, but not verifiable here

No address-range file this record reads, so nothing they did can be separated from an impersonator using their name.

| Agent | Claimed requests | Why |
| --- | --- | --- |
| Meta-ExternalAgent | 126,727 | no range file this record reads: cannot be told from an impersonator |
| Bytespider | 20,109 | no range file this record reads: cannot be told from an impersonator |
| Google-Extended | 49 | no range file this record reads: cannot be told from an impersonator |
| Diffbot | 2 | no range file this record reads: cannot be told from an impersonator |
| SemrushBot | 30,353 | no range file this record reads: cannot be told from an impersonator |
| DataForSeoBot | 16,743 | no range file this record reads: cannot be told from an impersonator |
| anthropic-ai | 4 | no range file this record reads: cannot be told from an impersonator |
| Applebot-Extended | 2 | no range file this record reads: cannot be told from an impersonator |
| Timpibot | 4 | no range file this record reads: cannot be told from an impersonator |
| YouBot | 2,524 | no range file this record reads: cannot be told from an impersonator |
| Scrapy | 18 | no range file this record reads: cannot be told from an impersonator |
| cohere-ai | 62 | no range file this record reads: cannot be told from an impersonator |
| FacebookBot | 1 | no range file this record reads: cannot be told from an impersonator |

Method

## Where the counts come from

Every request to this site that names a tracked agent is checked against the operator’s published ranges and counted by path. Other sites add what they found through [Visitor Resolve](/glossary/visitor-resolve) (the site_checks column in the record). The monthly population view is the [Agent Traffic Index](/api/v1/traffic-index).

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [Conduct: Claude searchbot](/agents/conduct/claude-searchbot) | **Blog:** [AI crawler IP ranges from 18 operator lists, in one file](/blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file) | **Glossary:** [FCrDNS](/glossary/fcrdns) · [Semantic cache](/glossary/semantic-cache) · [Entity-first content](/glossary/entity-first-content) · [Reduced motion](/glossary/reduced-motion) · [revocation delta](/glossary/revocation-delta) · [Sitemap directive](/glossary/sitemap-directive)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
