Wildcard rule
A robots path pattern using `*` to match a sequence of characters.
What Wildcard rule means
Implementations should be tested because extensions and encoding details can change apparent matches.
Where Wildcard rule comes up on this site
- How to block AI crawlers like GPTBot and ClaudeBotAdd a named group to restrict one crawler and that crawler stops seeing every wildcard rule: User-agent: * Disallow: /wp-admin/ Disallow: /checkout/ User-agent: GPTBot Disallow: /pricing/ GPTBot is now kept out of /pricing/ and allowed into /wp-admin/ and…
- Naming a crawler cancels your * rules: Semrush's robots.txtFor each named group, compare its rules with the * group: any wildcard rule not repeated inside the named group does not apply to that crawler.
- Only 6% of websites block an AI crawler: the full breakdownIf you block a crawler for some paths rather than all of them, its group has to repeat every wildcard rule you still want applied, as the guide to user-agent groups explains.
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Sitemap directive · Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Wildcard rule
What is Wildcard rule?
A robots path pattern using `*` to match a sequence of characters.
Which area of the glossary does Wildcard rule belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
← Longest-match rule · End-anchor rule →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.