Crawler identity
The operator and purpose attributed to an automated requester.
What Crawler identity means
A user-agent string asserts identity; verification requires network, DNS, or cryptographic evidence tied to the operator.
Where Crawler identity comes up on this site
- How to measure AI visibility yourself: three checksSo for each request claiming a crawler identity, check the source address against the range the operator publishes.
- robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlersIf the answers differ between the browser and the crawler identity, the crawler's answer is the one that matters.
- agents.md: AI agent instructions you may not have writtenSo ours states what is here — the scan, the aggregate data, the policy, the crawler identity — and asks an agent for nothing.
- Release notes 3 Oct 2026: Resolve, census, rulebookThe answer covers the crawl policy the site declares, what each crawler identity was actually served, the machine files, who the site says it is and whether outside sources confirm it, and any open findings.
Definitions that name Crawler identity
verified crawler · delivery comparison
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Sitemap directive · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Crawler identity
What is Crawler identity?
The operator and purpose attributed to an automated requester.
Which area of the glossary does Crawler identity belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
← Sitemap directive · Product token →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.