CrawlCheck

Agent view · Guides · 2026-11-13

Block AI crawlers in nginx, Apache and Caddy (tested)

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/block-ai-crawlers-nginx-apache-caddy: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/block-ai-crawlers-nginx-apache-caddy>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes12,358
words1,731
sha-256d1fefad84252fd2aafa71cda155d60c15b13d5322ab8f10ea79c863305e60c8d
repr-digestsha-256=:0f762EJS/SqvpxzaFV1gwVsT1TIquPEOp5yGMwXmDI0=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795180553;expires=1795184153;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="etsS0nzzRYweejtHgbeLXw";tag="crawlcheck-page"
signaturesig1=:cjyT+8/pwYOxu9znQslCbBnm8PUQ7W4waJuvbDc5gC8pii1XtP1FhqQB9CP/egl+tJqby1tR9ND4JvVOQNMqBQ==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author)

# Block AI crawlers in nginx, Apache and Caddy (tested)

Server rules that answer 403 to AI crawlers by name, for nginx, Apache and Caddy, with the test we ran on real servers and the two mistakes that make such rules silently fail.

To block AI crawlers at the server, match the User-Agent header against a list of crawler names and return 403. In nginx use a map in the http block and an if in the server block; in Apache a RewriteCond with the NC flag; in Caddy a header_regexp matcher with an inline (?i) flag. We tested all three on real servers: named crawlers got 403, Googlebot, Bingbot and browsers got 200.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- How the rule works
- nginx
- Apache
- Caddy
- The test we ran
- Two ways these rules fail quietly
- Before you block everything

[robots.txt](/glossary/robots-txt) asks. A server rule refuses. If you have decided a crawler should not get your pages, and especially if it is one of the fetchers whose operators say they [may ignore robots.txt](/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers), the server is where that decision takes effect.

The rules below are short examples. The full, current lists are generated from our [crawler catalogue](/ai-crawlers) and published on the [ready-made policies page](/robots-policies) in three sets: training and corpus crawlers, every AI crawler, and SEO and ad crawlers.

## How the rule works #

## nginx #

The map goes in the `http` block and the test in each `server` block. A map is evaluated once per request and keeps the list in one place.

```
map $http_user_agent $cc_block {
 default 0;
 "~*(GPTBot|ClaudeBot|CCBot|Bytespider)" 1;
}

server {
 if ($cc_block) { return 403; }
}
```

## Apache #

In `.htaccess` or a virtual host, with mod_rewrite enabled. The `NC` flag makes the match case-insensitive.

```
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider) [NC]
RewriteRule .* - [F,L]
```

## Caddy #

Inside a site block. Caddy's `header_regexp` is case-sensitive by default, so the pattern carries an inline `(?i)`.

```
@cc_block header_regexp User-Agent (?i)(GPTBot|ClaudeBot|CCBot|Bytespider)
respond @cc_block 403
```

## The test we ran #

On 13 November 2026 we loaded the published "every AI crawler" rule into nginx 1.24, Apache 2.4 and Caddy 2.8 and sent eight requests to each.

| [User agent](/glossary/user-agent) sent | Expected | nginx | Apache | Caddy |
| --- | --- | --- | --- | --- |
| GPTBot (full string) | 403 | 403 | 403 | 403 |
| chatgpt-user (lower case) | 403 | 403 | 403 | 403 |
| archive.org_bot | 403 | 403 | 403 | 403 |
| meta-externalfetcher | 403 | 403 | 403 | 403 |
| Googlebot | 200 | 200 | 200 | 200 |
| bingbot | 200 | 200 | 200 | 200 |
| a desktop browser | 200 | 200 | 200 | 200 |
| archiveXorg_bot (near miss) | 200 | 200 | 200 | 200 |

The lower-case row checks case handling. The last row checks that the dot in `archive.org_bot` is escaped: an unescaped dot matches any character and would block names you never listed. We also publish a Cloudflare rule expression; we have not loaded that one into a live zone, so treat it as untested.

## Two ways these rules fail quietly #

**Case.** Crawlers do not always send their name the way the documentation prints it. A case-sensitive match on `ChatGPT-User` lets `chatgpt-user` through, and nothing in your logs says the rule missed.

**Forged names.** A user agent is text anyone can send. A name-based rule stops crawlers that identify themselves; it does nothing about a scraper using a browser's name, and it will block a real person who sends a crawler's name by accident. Where the operator publishes address ranges, they are the stronger check. We mirror them in [one file](/ai-crawlers/ip-ranges.json).

## Before you block everything #

Blocking training crawlers and blocking answer engines are different decisions. The first changes what future models learn. The second removes you from answers people are reading now. The three published sets keep them apart so you can choose one without the other, and none of them touches classic search crawlers.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fblock-ai-crawlers-nginx-apache-caddy%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fblock-ai-crawlers-nginx-apache-caddy%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Block%20AI%20crawlers%20in%20nginx%2C%20Apache%20and%20Caddy%20(tested))[Email](mailto:?subject=Block%20AI%20crawlers%20in%20nginx%2C%20Apache%20and%20Caddy%20(tested)&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fblock-ai-crawlers-nginx-apache-caddy)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How do I block GPTBot in nginx?

Add a map in the http block that sets a variable to 1 when $http_user_agent matches GPTBot case-insensitively, then return 403 in the server block when that variable is set.

### How do I block AI crawlers in Apache?

With mod_rewrite: a RewriteCond on %{HTTP_USER_AGENT} listing the crawler names with the NC flag, followed by RewriteRule .* - [F,L], in .htaccess or a virtual host.

### Why does my Caddy rule not block a crawler?

Caddy's header_regexp matcher is case-sensitive. Add an inline (?i) flag at the start of the pattern so it matches the name in any case.

### Is blocking by user agent enough?

It stops crawlers that name themselves. A user agent can be forged, so where an operator publishes its address ranges, checking the address is the stronger test.

### Will these rules block Googlebot or Bingbot?

No. The published sets list AI training crawlers, AI answer engines, and SEO and ad crawlers. In our test Googlebot, Bingbot and a normal browser all received 200.

## Related findings

[2026-11-13 Our mirror relayed real crawlers and inflated forgery counts A mirror we ran relayed Meta and Amazon crawler requests to crawlcheck.io, and we counted them as failed checks. Closed 13 Novembe](/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts)[2026-11-13 Release notes: 7 to 13 November 2026 CrawlCheck release notes for 7 to 13 November 2026: block rules for four servers, crawler IP ranges in one file, catalogue correct](/blog/release-notes-7-to-13-november-2026)[2026-08-28 Does llms.txt work? What is known and how to test it Does llms.txt work? What is verifiable, what we measured on our own site, and the server-log test with IP verification that answer](/guides/does-llms-txt-actually-work)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Next.js](/fix/nextjs) · [Conduct: Storebot google](/agents/conduct/storebot-google) | **Datasets:** [Oregon](/datasets/state/oregon) · [Technologies: Optimizely](/datasets/technologies/optimizely) · [Problems: Declared sitemap refused](/datasets/problems/declared-sitemap-refused) | **Blog:** [A newsletter form that lost every sign-up](/blog/a-newsletter-form-that-lost-every-sign-up) | **Glossary:** [Manifest tampering](/glossary/manifest-tampering) · [Answer synthesis](/glossary/answer-synthesis)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/block-ai-crawlers-nginx-apache-caddy.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.