CrawlCheck Skip to content

Menu

For sites

Free A–F report, every problem named

Register your site

Free. Every agent that asks is told who runs it

Daily checks from $29 a month, an alert the day anything breaks

We do the fixes for $749 and prove them

Listing check, compare two sites, the browser extension, YouTube

Register your site

Prove who runs it.

Registered sites answer every agent that asks: who runs this, what it declares, verified by a fetch. Free.

Data

US sites by state, category, grade and platform

AI consent data

What every site tells AI crawlers, signed

Websites by problem

One lead list per problem we find

AI crawler census

The top 1,000 sites, read daily, open CC BY data

Decisions, scans and crawler visits, counted

What registered sites declare, verified by a fetch

SEO and AI tools, measured

The vendors that measure sites, measured by the same rules

Lead lists

Sites sorted by what they serve.

Pick a state, category, grade or platform. See the live count and sample rows before you buy.

For agents

MCP, curl, Python, OpenAI Agents, Claude tool use

The signed answer about any domain, for a person

Capability registry

What sites say agents may do, and what was verified

How each named agent behaves on sites

Before you act

One signed answer, no key.

Read, cite, connect or transact: ask first, verify the receipt offline.

Start in a minute

Developers

Every endpoint, with a worked example

Resolve protocol

What the signed answer holds, how to verify it

npm and PyPI, with the offline verifier

The Ed25519 keys every answer is signed with

Every finding the measurement can raise

Learn

The scan, step by step, and what each grade means

Guides, findings and the glossary

Who sends each agent and what it reads

Pricing Sign in

Glossary · Crawling and indexing

Wildcard rule

A robots path pattern using `*` to match a sequence of characters.

What Wildcard rule means

Implementations should be tested because extensions and encoding details can change apparent matches.

Where Wildcard rule comes up on this site

Our robots.txt check misread a full block as a broken ruleThe fix The check now asks whether each wildcard rule is covered by any rule the named group declares, the way a crawler matches paths: a rule covers every path that starts with it, a * matches any run of characters, and a $ anchors the end.
Only 6% of websites block an AI crawler: the full breakdownIf you block a crawler for some paths rather than all of them, its group has to repeat every wildcard rule you still want applied, as our correction on that check explains.

Related terms in Crawling and indexing

How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth

Questions about Wildcard rule

What is Wildcard rule?

A robots path pattern using `*` to match a sequence of characters.

Which area of the glossary does Wildcard rule belong to?

Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.

← Longest-match rule · End-anchor rule →

See it in the full glossary · 751 terms across 19 areas.