Glossary · Crawlers and access
agent conduct record
A per-agent record of how a crawler actually behaved on a measured property: whether it fetched paths the site's robots.txt disallows, its busiest minute against the site's own rate limit, whether it revalidates with conditional requests, and how often its name is forged by others. Only requests verified by source address count, so an impersonator's behaviour never lands on the agent.
What agent conduct record means
A per-agent record of how a crawler actually behaved on a measured property: whether it fetched paths the site's robots.txt disallows, its busiest minute against the site's own rate limit, whether it revalidates with conditional requests, and how often its name is forged by others. Only requests verified by source address count, so an impersonator's behaviour never lands on the agent. It can say what an agent did on the site that measured it; it cannot generalise to every site, and it withholds a verdict below a stated number of verified requests.
What agent conduct record can and cannot support
| It can support | It cannot support |
|---|---|
| A per-agent record of how a crawler actually behaved on a measured property: whether it fetched paths the site's robots.txt disallows, its busiest minute against the site's own rate limit, whether it revalidates with conditional requests, and how often its name is forged by others. | Only requests verified by source address count, so an impersonator's behaviour never lands on the agent. It can say what an agent did on the site that measured it; it cannot generalise to every site, and it withholds a verdict below a stated number of verified requests. |
Terms this definition uses
robots.txt · rate limit · Conditional request
Definitions that name agent conduct record
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about agent conduct record
What is agent conduct record?
A per-agent record of how a crawler actually behaved on a measured property: whether it fetched paths the site's robots.txt disallows, its busiest minute against the site's own rate limit, whether it revalidates with conditional requests, and how often its name is forged by others. Only requests verified by source address count, so an impersonator's behaviour never lands on the agent.
What does agent conduct record not show or guarantee?
Only requests verified by source address count, so an impersonator's behaviour never lands on the agent. It can say what an agent did on the site that measured it; it cannot generalise to every site, and it withholds a verdict below a stated number of verified requests.
Which area of the glossary does agent conduct record belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← agent arrival · agent traffic index →
See it in the full glossary · 688 terms across 19 areas.