CrawlCheck Skip to content

Menu

For humans

Check and fix

Register your site

Free. Every agent that asks is told who runs it

Free A–F report, every problem named

Checked twice a day from $29 a month, with an alert when a check finds a regression

We do the fixes for $100 and prove them

Executive report

A two-page PDF per site, monthly or weekly

The scan, step by step, and what each grade means

Free tools

Every check, no account

Your homepage as Chrome and as each AI crawler

robots.txt for AI crawlers

What each crawler is allowed, and what it got

Read the way crawlers ask for it

Is your business record right where agents look?

Compare two sites

Side by side, same measurement

Googlebot check

Is this IP from your logs really Google?

Entity map validator

entitymap.json against your structured data

Fix it on your platform

Browser extension

Grades as you browse

iPhone and Android

Register your site

Prove who runs it.

Registered sites answer every agent that asks: who runs this, what it declares, verified by a fetch. Free.

Datasets

For agents

Build

MCP, curl, Python, OpenAI Agents, LangChain, CrewAI, Vercel AI

Every endpoint, with a worked example

npm and PyPI, with the offline verifier

An approved server allowed, a rewritten tool refused

CrawlCheck in CI

Fail a pull request that breaks crawler access

Preflight rules and tests in your repository

Ask before acting

The signed answer about any domain, for a person

Resolve protocol

What the signed answer holds, how to verify it

Capability registry

What sites say agents may do, and what was verified

How each named agent behaves on sites

MCP tool changes

Servers whose tools changed after approval

Enterprise pilot

30 days of signed preflight for your agents

Verify it yourself

Verify a record

Check any answer on your own machine

The Ed25519 keys every answer is signed with

Every finding the measurement can raise

Canonical numbers

The definition and every public count, with sources and as_of

Run the suite against your own implementation

Proof and timestamps

How every record is sealed and anchored

Scans, signing, seals and a signed incident log

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline.

Start in a minute

Research

Live data

AI crawler census

The top 1,000 sites, read daily, open CC BY data

AI crawler radar

What the top 10,000 sites say and serve

Agent Traffic Index

Verified and forged agent requests, monthly

Decisions, scans and crawler visits, counted

Agent safety benchmark

Do agents stop before acting?

Every city a scan was started from

Crawlers and vendors

Who sends each agent and what it reads

Crawlers compared

GPTBot vs ClaudeBot and more, two at a time

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

What registered sites declare, verified by a fetch

Certified sites

Every live machine-readability certificate

Evidence for disputes

A dated, signed record of what a site served

Reports and method

Four recurring reports, each signed with a feed

Exclusive findings

Measured only here

How we read the data

The rules, what we never cite, how to report an error

Six lessons and a certificate

MCP tool changes

How often public MCP servers change the tools agents approved

AI audit tools, Round 2

74 tools against 63 planted faults

AI audit tools, tested

Twelve tools against ten planted faults

Submit a finding

Your study, published with credit

Everything we got wrong, dated

The census

What the top sites serve AI crawlers.

Read daily from our own network, open CC BY data, every figure with its denominator.

Read the census

Library Pricing Sign in

Glossary · Indexing and discovery

case consistency

Whether the site's own links and sitemap entries agree on how a URL is written: lowercase paths, one trailing-slash form, one host form, no navigation through query strings. To a crawler /About and /about are two URLs, and a site that links to both has published every page twice.

What case consistency does not show

Scored from the hrefs and sitemap entries the site itself serves, never from the paths CrawlCheck chose to request.

Terms this definition uses

Definitions that name case consistency

faceted navigation

Related terms in Indexing and discovery

Whether a page can be found and kept, separately from whether it can be fetched.

nosnippet · max-snippet · max-image-preview · unavailable_after · noimageindex · index coverage · canonical · noindex · X-Robots-Tag · orphan page · E-E-A-T · E-E-A-T proxies · topical authority · Soft 404 · accessibility structure · landmark region · accessible name · skip link · main landmark · alt attribute · form control name · duplicate id · IndexNow · faceted navigation · thin content · duplicate content · near-duplicate · boilerplate · MinHash · shingle · dead anchor · placeholder content · image placeholder · alt text · EXIF

Questions about case consistency

What is case consistency?

Whether the site's own links and sitemap entries agree on how a URL is written: lowercase paths, one trailing-slash form, one host form, no navigation through query strings. To a crawler /About and /about are two URLs, and a site that links to both has published every page twice.

What does case consistency not show or guarantee?

Scored from the hrefs and sitemap entries the site itself serves, never from the paths CrawlCheck chose to request.

Which area of the glossary does case consistency belong to?

Indexing and discovery: Whether a page can be found and kept, separately from whether it can be fetched.

← Soft 404 · accessibility structure →

See it in the full glossary · 772 terms across 19 areas.

Free browser extension

See this on any site you visit, in one click.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Add to Chrome — free