Crawl-delay

What Crawl-delay does not show

Where Crawl-delay comes up on this site

• Naming a crawler in robots.txt drops your Disallow rulesFirst, a parser that closes a group at any unfamiliar line, such as Crawl-delay or Content-Signal , will attach the following rules to the wrong group; keep a group open across every directive until the next User-agent line that follows a rule.

Terms this definition uses

robots.txt

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Sitemap directive · Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Crawl-delay

What is Crawl-delay?

A non-standard robots.txt directive requesting time between crawler requests.

What does Crawl-delay not show or guarantee?

Support varies, and RFC 9309 does not define it.

Which area of the glossary does Crawl-delay belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← Crawl rate  ·  Robots group merging →

See it in the full glossary · 772 terms across 19 areas.