AI crawlers · Search indexes

`Googlebot`

Googlebot is the search index crawler run by Google: Google Search / AI Overviews. Its operator publishes the IP ranges it crawls from (https://developers.google.com/static/crawling/ipranges/common-crawlers.json), so a request carrying the name can be checked against that list.

User-agent token	`Googlebot`
Operator	Google
What it does	Google Search / AI Overviews
Kind	search index crawler
Can a request be verified?	Yes — published ip feed
Verification source	https://developers.google.com/static/crawling/ipranges/common-crawlers.json
In a CrawlCheck scan	Sent as a request on every scan; a refusal is scored

Allow or block it in robots.txt

User-agent: Googlebot
Disallow: /

User-agent: Googlebot
Allow: /

Add a group that names it: "User-agent: Googlebot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Googlebot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

Indirectly. It feeds Google Search / AI Overviews. A refusal removes pages from that index, and most AI answer surfaces still draw on search indexes.

What it actually did on crawlcheck.io

Verified requests	8,244
Requests using its name from outside its operator's ranges	156 (1.9% of claims)
robots.txt fetched	127 times
Fetches of paths robots.txt disallows	0
Observed	2026-08-13 to 2026-10-30

Measured on this site’s own traffic, verified against the operator’s published ranges. Every agent’s record: agent conduct.

Other agents from Google

Google-CloudVertexBot — Vertex AI grounding
Google-Extended — Gemini training
Googlebot-Image — Google image crawler
Googlebot-News — Google News
Googlebot-Video — Google video crawler
Storebot-Google — Google Shopping
Mediapartners-Google — Google AdSense
APIs-Google — Google push delivery
GoogleOther — Google non-search fetch
AdsBot-Google — Google Ads quality
Gemini-Deep-Research — Gemini research agent
Google-Agent — Google user-triggered agent, acts on the web
Google-GeminiNotebook — Gemini Notebook user-supplied URL fetch
GoogleOther-Image — Google common crawler, public image URLs
GoogleOther-Video — Google common crawler, public video URLs

See what Googlebot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Googlebot

QHow do I block Googlebot in robots.txt?

QHow can I tell if a request from Googlebot is real?

Its operator publishes the IP ranges it crawls from (https://developers.google.com/static/crawling/ipranges/common-crawlers.json), so a request carrying the name can be checked against that list.

QDoes blocking Googlebot affect whether my site appears in AI answers?

Indirectly. It feeds Google Search / AI Overviews. A refusal removes pages from that index, and most AI answer surfaces still draw on search indexes.

QDoes CrawlCheck test my site as Googlebot?

Yes. Every CrawlCheck scan fetches the homepage as Googlebot and compares the response with what a browser receives. A refusal of it counts toward the grade, because it costs the site visibility.