AI crawlers · Training crawlers

`Google-CloudVertexBot`

Google-CloudVertexBot is the training crawler run by Google: Vertex AI grounding. Its operator documents how to verify it: Google special-crawler ranges (shared feed) + reverse DNS to google.com. A reverse DNS lookup on the source address must return the operator's domain, and a forward lookup on that name must return the same address.

User-agent token	`Google-CloudVertexBot`
Operator	Google
What it does	Vertex AI grounding
Kind	training crawler
Can a request be verified?	Yes — reverse dns
Verification source	Google special-crawler ranges (shared feed) + reverse DNS to google.com
In a CrawlCheck scan	Read from robots.txt; reported, never scored

Allow or block it in robots.txt

User-agent: Google-CloudVertexBot
Disallow: /

User-agent: Google-CloudVertexBot
Allow: /

Add a group that names it: "User-agent: Google-CloudVertexBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Google-CloudVertexBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

Other agents from Google

Googlebot — Google Search / AI Overviews
Google-Extended — Gemini training
Googlebot-Image — Google image crawler
Googlebot-News — Google News
Googlebot-Video — Google video crawler
Storebot-Google — Google Shopping
Mediapartners-Google — Google AdSense
APIs-Google — Google push delivery
GoogleOther — Google non-search fetch
AdsBot-Google — Google Ads quality
Gemini-Deep-Research — Gemini research agent
Google-Agent — Google user-triggered agent, acts on the web
Google-GeminiNotebook — Gemini Notebook user-supplied URL fetch
GoogleOther-Image — Google common crawler, public image URLs
GoogleOther-Video — Google common crawler, public video URLs

See what Google-CloudVertexBot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about Google-CloudVertexBot

QHow do I block Google-CloudVertexBot in robots.txt?

QHow can I tell if a request from Google-CloudVertexBot is real?

Its operator documents how to verify it: Google special-crawler ranges (shared feed) + reverse DNS to google.com. A reverse DNS lookup on the source address must return the operator's domain, and a forward lookup on that name must return the same address.

QDoes blocking Google-CloudVertexBot affect whether my site appears in AI answers?

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

QDoes CrawlCheck test my site as Google-CloudVertexBot?

Not as a request. CrawlCheck reads the rules your robots.txt sets for Google-CloudVertexBot and reports them. Rules for it are reported and never scored, because refusing it is a policy choice.