CrawlCheck Skip to content

Menu

For humans

Check and fix

Register your site

Free. Every agent that asks is told who runs it

Free A–F report, every problem named

Checked twice a day from $29 a month, with an alert when a check finds a regression

We do the fixes for $100 and prove them

Executive report

A two-page PDF per site, monthly or weekly

The scan, step by step, and what each grade means

Free tools

Every check, no account

Your homepage as Chrome and as each AI crawler

robots.txt for AI crawlers

What each crawler is allowed, and what it got

Read the way crawlers ask for it

Is your business record right where agents look?

Compare two sites

Side by side, same measurement

Googlebot check

Is this IP from your logs really Google?

Entity map validator

entitymap.json against your structured data

Fix it on your platform

Browser extension

Grades as you browse

iPhone and Android

Register your site

Prove who runs it.

Registered sites answer every agent that asks: who runs this, what it declares, verified by a fetch. Free.

Datasets

For agents

Build

MCP, curl, Python, OpenAI Agents, LangChain, CrewAI, Vercel AI

Every endpoint, with a worked example

npm and PyPI, with the offline verifier

An approved server allowed, a rewritten tool refused

CrawlCheck in CI

Fail a pull request that breaks crawler access

Preflight rules and tests in your repository

Ask before acting

The signed answer about any domain, for a person

Resolve protocol

What the signed answer holds, how to verify it

Capability registry

What sites say agents may do, and what was verified

How each named agent behaves on sites

MCP tool changes

Servers whose tools changed after approval

Enterprise pilot

30 days of signed preflight for your agents

Verify it yourself

Verify a record

Check any answer on your own machine

The Ed25519 keys every answer is signed with

Every finding the measurement can raise

Canonical numbers

The definition and every public count, with sources and as_of

Run the suite against your own implementation

Proof and timestamps

How every record is sealed and anchored

Scans, signing, seals and a signed incident log

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline.

Start in a minute

Research

Live data

AI crawler census

The top 1,000 sites, read daily, open CC BY data

AI crawler radar

What the top 10,000 sites say and serve

Agent Traffic Index

Verified and forged agent requests, monthly

Decisions, scans and crawler visits, counted

Agent safety benchmark

Do agents stop before acting?

Every city a scan was started from

Crawlers and vendors

Who sends each agent and what it reads

Crawlers compared

GPTBot vs ClaudeBot and more, two at a time

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

What registered sites declare, verified by a fetch

Certified sites

Every live machine-readability certificate

Evidence for disputes

A dated, signed record of what a site served

Reports and method

Four recurring reports, each signed with a feed

Exclusive findings

Measured only here

How we read the data

The rules, what we never cite, how to report an error

Six lessons and a certificate

MCP tool changes

How often public MCP servers change the tools agents approved

AI audit tools, Round 2

74 tools against 63 planted faults

AI audit tools, tested

Twelve tools against ten planted faults

Submit a finding

Your study, published with credit

Everything we got wrong, dated

The census

What the top sites serve AI crawlers.

Read daily from our own network, open CC BY data, every figure with its denominator.

Read the census

Library Pricing Sign in

Glossary · Content architecture and SEO

Taxonomy

A controlled hierarchy of categories used to classify content.

What Taxonomy means

It improves consistency while becoming harmful when categories overlap, remain empty, or generate unlimited URLs.

Related terms in Content architecture and SEO

How pages, links and intent are organised so an entity is findable, and what a link or a structure can and cannot transfer.

lang attribute · x-default · Search intent · Query class · Navigational query · Informational query · Transactional query · Commercial investigation · Entity-first content · Information architecture · Ontology · Topic cluster · Pillar page · Content hub · Semantic HTML · Heading hierarchy · Internal link · Anchor text · Link context · Deep link · Broken link · Redirected internal link · Link equity · PageRank · Dofollow · Nofollow · Sponsored link · User-generated link · External link · Inbound link · Referring domain · Link neighborhood · Orphaned entity · Content decay · Content refresh · Historical optimization · answer block · apex domain · alias domain · parked domain · local pack · Google Business Profile · data aggregator · Data Axle

Questions about Taxonomy

What is Taxonomy?

A controlled hierarchy of categories used to classify content.

Which area of the glossary does Taxonomy belong to?

Content architecture and SEO: How pages, links and intent are organised so an entity is findable, and what a link or a structure can and cannot transfer.

← Information architecture · Ontology →

See it in the full glossary · 772 terms across 19 areas.

Free browser extension

See this on any site you visit, in one click.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Add to Chrome — free