verified crawler
A request whose claimed crawler identity is confirmed by checking the source address against the operator's own published IP ranges.
What verified crawler does not show
Reverse DNS and user-agent matching are weaker: only the operator's feed is authoritative.
Where verified crawler comes up on this site
- How to measure AI visibility yourself: three checksPer repetition: the date, what was asked, which surface answered, and whether you appeared, alongside the step-one results and the verified crawler counts for the same date, so a change in one can be read against the others.
- AI crawler monitoring vs uptime monitors: what each one catchesThe only direct evidence of what a verified crawler received is the site’s own access log.
- Our crawler counts missed about 45% of hitsWhat the counter did Each verified crawler hit incremented a key named for the crawler and the site, tel:<bot>:<host> , plus a shared daily budget key.
Terms this definition uses
Definitions that name verified crawler
crawler classification · rate limit
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about verified crawler
What is verified crawler?
A request whose claimed crawler identity is confirmed by checking the source address against the operator's own published IP ranges.
What does verified crawler not show or guarantee?
Reverse DNS and user-agent matching are weaker: only the operator's feed is authoritative.
Which area of the glossary does verified crawler belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← user-triggered fetch · forged crawler identity →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.