Product token

What Product token means

RFC 9309 limits its characters and expects it to appear within the crawler’s HTTP identification string.

Where Product token comes up on this site

• Naming a crawler in robots.txt drops your Disallow rulesThe rule: one crawler, one group RFC 9309, the Robots Exclusion Protocol, says a crawler must obey the group whose User-agent line best matches its product token.

Terms this definition uses

robots.txt · Identification string

Definitions that name Product token

Robots group merging

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Sitemap directive · Crawler identity · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Product token

What is Product token?

The crawler name used to select a robots.txt group.

Which area of the glossary does Product token belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← Crawler identity  ·  Identification string →

See it in the full glossary · 772 terms across 19 areas.