CrawlCheck Skip to content

Menu

For humans

Check and fix

Register your site

Free. Every agent that asks is told who runs it

Free A–F report, every problem named

Checked twice a day from $29 a month, with an alert when a check finds a regression

We do the fixes for $100 and prove them

Executive report

A two-page PDF per site, monthly or weekly

The scan, step by step, and what each grade means

Free tools

Every check, no account

Your homepage as Chrome and as each AI crawler

robots.txt for AI crawlers

What each crawler is allowed, and what it got

Read the way crawlers ask for it

Is your business record right where agents look?

Compare two sites

Side by side, same measurement

Googlebot check

Is this IP from your logs really Google?

Entity map validator

entitymap.json against your structured data

Fix it on your platform

Browser extension

Grades as you browse

iPhone and Android

Register your site

Prove who runs it.

Registered sites answer every agent that asks: who runs this, what it declares, verified by a fetch. Free.

Datasets

For agents

Build

MCP, curl, Python, OpenAI Agents, LangChain, CrewAI, Vercel AI

Every endpoint, with a worked example

npm and PyPI, with the offline verifier

An approved server allowed, a rewritten tool refused

CrawlCheck in CI

Fail a pull request that breaks crawler access

Preflight rules and tests in your repository

Ask before acting

The signed answer about any domain, for a person

Resolve protocol

What the signed answer holds, how to verify it

Capability registry

What sites say agents may do, and what was verified

How each named agent behaves on sites

MCP tool changes

Servers whose tools changed after approval

Enterprise pilot

30 days of signed preflight for your agents

Verify it yourself

Verify a record

Check any answer on your own machine

The Ed25519 keys every answer is signed with

Every finding the measurement can raise

Canonical numbers

The definition and every public count, with sources and as_of

Run the suite against your own implementation

Proof and timestamps

How every record is sealed and anchored

Scans, signing, seals and a signed incident log

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline.

Start in a minute

Research

Live data

AI crawler census

The top 1,000 sites, read daily, open CC BY data

AI crawler radar

What the top 10,000 sites say and serve

Agent Traffic Index

Verified and forged agent requests, monthly

Decisions, scans and crawler visits, counted

Agent safety benchmark

Do agents stop before acting?

Every city a scan was started from

Crawlers and vendors

Who sends each agent and what it reads

Crawlers compared

GPTBot vs ClaudeBot and more, two at a time

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

What registered sites declare, verified by a fetch

Certified sites

Every live machine-readability certificate

Evidence for disputes

A dated, signed record of what a site served

Reports and method

Four recurring reports, each signed with a feed

Exclusive findings

Measured only here

How we read the data

The rules, what we never cite, how to report an error

Six lessons and a certificate

MCP tool changes

How often public MCP servers change the tools agents approved

AI audit tools, Round 2

74 tools against 63 planted faults

AI audit tools, tested

Twelve tools against ten planted faults

Submit a finding

Your study, published with credit

Everything we got wrong, dated

The census

What the top sites serve AI crawlers.

Read daily from our own network, open CC BY data, every figure with its denominator.

Read the census

Library Pricing Sign in

Glossary · Crawling and indexing

Crawl frontier

The working collection of discovered URLs waiting for crawl decisions.

What Crawl frontier does not show

Presence on the frontier proves discovery, not that the URL will be fetched.

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Sitemap directive · Crawler identity · Product token · Identification string · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Crawl frontier

What is Crawl frontier?

The working collection of discovered URLs waiting for crawl decisions.

What does Crawl frontier not show or guarantee?

Presence on the frontier proves discovery, not that the URL will be fetched.

Which area of the glossary does Crawl frontier belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← Identification string · Crawl queue →

See it in the full glossary · 772 terms across 19 areas.

Free browser extension

See this on any site you visit, in one click.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Add to Chrome — free