CrawlCheck Skip to content

Menu

How it works

For agents

Quickstart: ask before you act

MCP, curl, Python, OpenAI Agents, Claude tool use, LangChain

Every endpoint, with a worked example from production

Resolve protocol

The signed answer: what it holds, how to verify it

SDKs: npm and PyPI

Typed clients, the offline verifier, conformance fixtures

Capability registry

What sites declare agents can do, and what was verified

How each named agent behaves on sites, from verified requests

Every finding the measurement can raise, with its revision

The Ed25519 keys every answer is signed with

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline, keep it with the action.

Start in a minute

For sites

The checks behind the audit, one at a time

Is your business record right where agents look?

Compare two sites

Two domains, side by side, same measurement

What search and answer engines read from a video

Browser extension

The second vantage point: from your own address, after JavaScript

The record on your phone, with receipts

Your site

Measured as agents see it.

A free A–F audit with every defect named, no account. Fix it, then prove it held.

Learn

Findings from the measurement, with the numbers

Every term the product uses, defined

Who sends each agent, what it reads, how to verify it

Evidence

The public census and the daily snapshot, CC BY 4.0

Proof and timestamps

How every record is sealed and anchored

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

Registry: media kits, llms.txt, entity graphs

Verified machine-readable profiles of real organisations

Pricing Sign in

Glossary · Crawling and indexing

Crawl-delay

A non-standard robots.txt directive requesting time between crawler requests.

What Crawl-delay does not show

Support varies, and RFC 9309 does not define it.

Where Crawl-delay comes up on this site

robots.txt user-agent groups: why named bots skip your rulesFirst, a parser that closes a group at any unfamiliar line, such as Crawl-delay or Content-Signal , will attach the following rules to the wrong group; keep a group open across every directive until the next User-agent line that follows a rule.

Terms this definition uses

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Crawl-delay

What is Crawl-delay?

A non-standard robots.txt directive requesting time between crawler requests.

What does Crawl-delay not show or guarantee?

Support varies, and RFC 9309 does not define it.

Which area of the glossary does Crawl-delay belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← Crawl rate · Robots group merging →

See it in the full glossary · 731 terms across 19 areas.