CrawlCheck Skip to content

Menu

For agents

The door: what an agent should know before it acts on a site

Quickstart: ask before you act

MCP, curl, Python, OpenAI Agents, Claude tool use, LangChain

The signed answer about any domain, laid out for a person

How each named agent behaves on sites, from verified requests

Capability registry

What sites declare agents can do, and what was verified

Verify a record

Check any answer or bundle on your own machine

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline, keep it with the action.

Start in a minute

For sites

The door: scan free, have it fixed, keep it fixed

The checks behind the audit, one at a time

Is your business record right where agents look?

Compare two sites

Two domains, side by side, same measurement

What search and answer engines read from a video

Browser extension

The second vantage point: from your own address, after JavaScript

The record on your phone, with receipts

Your site

Measured as agents see it.

A free A–F audit with every defect named, no account. Fix it, then prove it held.

Developers

The door: the API by lane, the specs, SDKs, keys and quotas

Every endpoint, with a worked example from production

Resolve protocol

The signed answer: what it holds, how to verify it

SDKs: npm and PyPI

Typed clients, the offline verifier, conformance fixtures

The Ed25519 keys every answer is signed with

Every finding the measurement can raise, with its revision

Run the suite against your own implementation

Data

The door: census, conduct, proof and datasets, each with a denominator

Signed decisions, scans and crawler visits, counted

Proof and timestamps

How every record is sealed and anchored

CC BY 4.0 snapshots and the archive

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

Registry: media kits, llms.txt, entity graphs

Verified machine-readable profiles of real organisations

Learn

The door: guides, findings and the glossary, from measurements

The scan, stage by stage, and what each grade means

How to check one thing, with the rate at which it fails

Dated cases from real scans, with the bytes behind them

Every term the product uses, defined

Who sends each agent, what it reads, how to verify it

Pricing Sign in

Glossary · Crawling and indexing

Robots cache

A crawler’s stored copy of robots.txt.

What Robots cache does not show

A corrected file may not affect requests until that copy expires or is refreshed.

Terms this definition uses

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Robots cache

What is Robots cache?

A crawler’s stored copy of robots.txt.

What does Robots cache not show or guarantee?

A corrected file may not affect requests until that copy expires or is refreshed.

Which area of the glossary does Robots cache belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← End-anchor rule · Robots parse error →

See it in the full glossary · 731 terms across 19 areas.