Host politeness
Delaying or limiting requests so one origin is not overloaded.
What Host politeness does not show
It is an operator behavior rather than a guarantee in robots.txt.
Terms this definition uses
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Host politeness
What is Host politeness?
Delaying or limiting requests so one origin is not overloaded.
What does Host politeness not show or guarantee?
Delaying or limiting requests so one origin is not overloaded. It is an operator behavior rather than a guarantee in robots.txt.
Which area of the glossary does Host politeness belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
← Crawl frequency · Crawl rate →
See it in the full glossary · 751 terms across 19 areas.