CrawlCheck

Glossary · Indexing and discovery

faceted navigation

Filter combinations, size, colour, price, sort order, that each generate a URL, so that a catalogue of a hundred products can expose millions of addresses. It is the ordinary-case cause of a crawler trap, and the usual reason crawl budget is spent on pages that all show the same items. The fix is a rule about which combinations are canonical, which are noindexed, and which are blocked; case consistency in the parameter names is the related failure, since ?Color= and ?color= are two more URLs.

Terms this definition uses

Product · Crawler trap · crawl budget · canonical · noindex · case consistency · Parameter

Indexing and discovery

Whether a page can be found and kept, separately from whether it can be fetched.

index coverage · canonical · noindex · X-Robots-Tag · orphan page · E-E-A-T · E-E-A-T proxies · topical authority · Soft 404 · case consistency · accessibility structure · landmark region · accessible name · skip link · main landmark · alt attribute · form control name · duplicate id · IndexNow

IndexNow

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.