Crawler identity
The operator and purpose attributed to an automated requester.
What Crawler identity means
A user-agent string asserts identity; verification requires network, DNS, or cryptographic evidence tied to the operator.
Where Crawler identity comes up on this site
- The AI crawler and agent list: what each one is for, and which can be verifiedAcross every full scan CrawlCheck has run, of sites serve an ordinary browser but refuse at least one answer-engine crawler identity, which is how a user-triggered fetch gets blocked without anyone deciding to block it.
- How to measure AI and agent visibility without trusting a vendor's scoreSo the method is: for each request claiming a crawler identity, check the source address against the operator's published range.
- robots.txt 404, 403, 429 and 503: what each tells a crawlerIf the answers differ between the browser and the crawler identity, the crawler's answer is the one that matters.
- Best AI web crawler on a budget: real costs comparedWhat none of them is built to do, and what CrawlCheck exists for, is fetch it once per AI crawler identity, compare what each one was served against a browser, read robots.txt and the AI opt-out signals per crawler, and tell you which of those differences…
- Best AI visibility tools compared: prices and what they trackThat is the gap CrawlCheck was built for: one free scan fetches the site as a browser and as each AI crawler identity, compares what each received, resolves robots.txt per crawler, and names the cause.
Definitions that name Crawler identity
verified crawler · delivery comparison
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Crawler identity
What is Crawler identity?
The operator and purpose attributed to an automated requester.
Which area of the glossary does Crawler identity belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
See it in the full glossary · 751 terms across 19 areas.