Allowlist
An explicit set of permitted values, operations, hosts, or paths.
What Allowlist means
It reduces ambiguity but becomes dangerous when broad wildcards silently expand it.
Where Allowlist comes up on this site
- Naming a crawler in robots.txt drops your Disallow rulesWe made this mistake ourselves: on sites we operate, the allowlist we deployed shadowed every Disallow rule for every crawler it named, and our own check graded the files clean because each named crawler could reach the homepage.
- Is your host blocking ClaudeBot? Group the 403s by IP firstWhy cache the files rather than allowlist the crawlers?
- Which AI crawlers can't be verified by IP range?Ranges move: refresh, don't copy An allowlist compiled from a feed six months ago lists addresses that were right six months ago.
- How much AI crawler traffic is forged? We check the IPsSites allowlist Googlebot in rate limiters and paywalls, and increasingly allowlist the AI crawlers they want to be cited by.
Related terms in Security and privacy
Where an agent, a tool or a page can be turned against its operator, and the controls that limit the damage without proving safety.
X-Frame-Options · frame-ancestors · Referrer-Policy · Permissions-Policy · Encrypted Client Hello · lookalike domain · homoglyph · incident switch · high-risk field · Indirect prompt injection · Direct prompt injection · Jailbreak · Data exfiltration · Sensitive information disclosure · Secret leakage · Insecure output handling · Excessive agency · Least privilege · Trust boundary · Input validation · Output validation · Denylist · SSRF · RCE · Credential scope · OAuth scope · API key · Bearer token · mTLS · Rate-limit policy · Audit log · PII · Data minimization · Purpose limitation · Retention period · Data residency · Tenant isolation · Threat model · DMARC · DMARC alignment · SPF · DKIM · MTA-STS · TLS-RPT · CAA record · DNSSEC · hard bounce
Questions about Allowlist
What is Allowlist?
An explicit set of permitted values, operations, hosts, or paths.
Which area of the glossary does Allowlist belong to?
Security and privacy: Where an agent, a tool or a page can be turned against its operator, and the controls that limit the damage without proving safety.
← Output validation · Denylist →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.