# How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate Limits

> A tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.

Source: https://iabig.com/guides/platforms/nginx-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

With nginx, everything is handled in the server configuration: you can serve robots.txt, deny a request based on its User-Agent, check that a crawler really comes from its vendor's addresses, limit its request rate and write it to a separate log. This guide assembles those pieces into a complete configuration built around a simple policy: refuse model training, let the search crawlers of ChatGPT, Claude and Perplexity through, and return a 403 to anything that borrows their names without coming from their IP ranges. We validated this configuration with `nginx -t` and then with real requests on nginx 1.30.6, using the official Windows build; the Linux paths shown (`/etc/nginx`, `/var/log/nginx`) are those of the Debian and Ubuntu packages and should be adjusted for your distribution.

## robots.txt first

Server-side blocking does not replace robots.txt: it is the file that crawlers which respect your choices actually read, and the only way to refuse training to Google (`Google-Extended`) or Apple (`Applebot-Extended`), which never send requests under those names. Put the file at the site root and make sure it always returns a 200, whatever the crawler:

```nginx
location = /robots.txt {
    # Served to everyone, including crawlers denied elsewhere
    try_files $uri =404;
    access_log off;   # optional: keeps the main log uncluttered
}
```

RFC 9309 specifies that a robots.txt that is unreachable because of a 5xx error counts as a full disallow, and a robots.txt denied with a 403 prevents the crawler from reading your refusal in it. The configuration below therefore exempts `/robots.txt` from every block.

## The http block configuration

This file goes in the `http` context, for example in `/etc/nginx/conf.d/`, which Debian and Ubuntu include by default. Each `map` costs nothing until its variable is used, since nginx evaluates it on demand.

```nginx
# /etc/nginx/conf.d/ai-crawlers.conf  (http context)

# 1. AI crawler announced by the User-Agent.
#    Expressions are tested in order; ~* = case-insensitive.
map $http_user_agent $ai_crawler {
    default                 "";
    "~*GPTBot"              GPTBot;
    "~*OAI-SearchBot"       OAI-SearchBot;
    "~*ChatGPT-User"        ChatGPT-User;
    "~*ClaudeBot"           ClaudeBot;
    "~*Claude-SearchBot"    Claude-SearchBot;
    "~*Claude-User"         Claude-User;
    "~*PerplexityBot"       PerplexityBot;
    "~*Perplexity-User"     Perplexity-User;
    "~*CCBot"               CCBot;
    "~*meta-externalagent"  meta-externalagent;
    "~*Bytespider"          Bytespider;
}

# 2. Crawlers denied server-side (here: training; adjust as needed)
map $ai_crawler $crawler_denied {
    default             0;
    GPTBot              1;
    ClaudeBot           1;
    CCBot               1;
    meta-externalagent  1;
    Bytespider          1;
}

# 3. Claimed vendor, for crawlers whose IP ranges are published
map $ai_crawler $claimed_vendor {
    default           "";
    GPTBot            openai;
    OAI-SearchBot     openai;
    ChatGPT-User      openai;
    ClaudeBot         anthropic;
    Claude-SearchBot  anthropic;
    Claude-User       anthropic;
    PerplexityBot     perplexity;
    Perplexity-User   perplexity;
}

# 4. Actual vendor based on the IP address (file generated by the script below)
geo $ip_vendor {
    default none;
    include /etc/nginx/ai-crawlers-ip.conf;
}

# 5. 1 = announced crawler, but IP outside its vendor's ranges
map "$claimed_vendor:$ip_vendor" $crawler_spoofed {
    default                  1;
    "~^:"                    0;   # no verifiable crawler announced
    "openai:openai"          0;
    "anthropic:anthropic"    0;
    "perplexity:perplexity"  0;
}

# 6. Final decision. robots.txt always stays readable.
map "$crawler_denied$crawler_spoofed:$uri" $ai_deny {
    default                  0;
    "~^..:/robots\.txt$"     0;   # must remain the first expression
    "~^1"                    1;   # crawler denied on principle
    "~^01"                   1;   # spoofed crawler
}

# 7. Rate limiting: only AI crawlers get a key.
#    An empty key is not counted.
map $ai_crawler $ai_limit_key {
    ""       "";
    default  $binary_remote_addr;
}
limit_req_zone $ai_limit_key zone=ai_crawlers:10m rate=1r/s;

# 8. Format of the dedicated log
log_format ai_crawlers '$remote_addr [$time_local] "$request" $status '
                     'crawler=$ai_crawler ip=$ip_vendor denied=$ai_deny "$http_user_agent"';
```

A few notes on reading it. In a `map`, the nginx documentation states that the search stops at the first match, with exact strings checked before regular expressions, which are themselves tested in the order they appear in the file. The pattern `~*GPTBot` does not match `ChatGPT-User` (which contains "GPT-U", not "GPTBot"), and `~*ClaudeBot` does not match `Claude-SearchBot`, so the order above is unambiguous. The `geo` module accepts IPv4 and IPv6 prefixes in CIDR notation. Finally, `limit_req_zone` ignores requests whose key is empty, so ordinary visitors are not rate-limited.

## The server block

```nginx
server {
    listen 443 ssl;
    server_name www.example.com;
    # ssl_certificate and ssl_certificate_key: unchanged, omitted here
    root /var/www/example;

    # An access_log declared here replaces those inherited from http:
    # redeclare the main log, then the dedicated log.
    access_log /var/log/nginx/access.log combined;
    access_log /var/log/nginx/ai-crawlers.log ai_crawlers if=$ai_crawler;

    # 1 request/s per address for AI crawlers, bursts of 5 tolerated
    limit_req zone=ai_crawlers burst=5 nodelay;
    limit_req_status 429;

    # Deny (training crawlers and impostors), except for robots.txt
    if ($ai_deny) {
        return 403;
    }

    location = /robots.txt {
        try_files $uri =404;
    }

    location / {
        try_files $uri $uri/ =404;
    }
}
```

The rewrite module's `if` and `return` directives are allowed in the `server` context; placed there, they apply to every URL, whichever `location` is selected afterwards. The `if=` parameter of `access_log` only writes the line when the variable is neither empty nor equal to "0", so `ai-crawlers.log` only contains requests from an announced AI crawler, whether accepted or denied. The 429 status set by `limit_req_status` (503 by default) clearly tells the crawler to slow down.

If the site sits behind a CDN or a load balancer, `$remote_addr` holds that intermediary's address: configure the `realip` module first (`set_real_ip_from`, `real_ip_header`), otherwise every genuine crawler will be classified as an impostor.

## Keeping the official IP ranges up to date

OpenAI, Anthropic and Perplexity publish their ranges in JSON files with the same format: a `prefixes` array in which each element contains `ipv4Prefix` or `ipv6Prefix`. The following script, which we ran against the live files in October 2026 (a little over 300 prefixes in total), produces the file included by `geo` and only reloads nginx if the configuration is still valid.

```bash
#!/usr/bin/env bash
# /usr/local/sbin/update-ai-crawlers-nginx.sh - requires curl and jq
set -euo pipefail
output=/etc/nginx/ai-crawlers-ip.conf
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

add() {  # $1 = vendor, $2 = URL of the official JSON file
  # -L: follow redirects if a list moves
  curl -fsSL --max-time 30 "$2" \
    | jq -er --arg e "$1" '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null) | "\(.) \($e);"' >> "$tmp"
}
add openai     https://openai.com/gptbot.json
add openai     https://openai.com/searchbot.json
add openai     https://openai.com/chatgpt-user.json
add anthropic  https://claude.com/crawling/bots.json
add perplexity https://www.perplexity.ai/perplexitybot.json
add perplexity https://www.perplexity.ai/perplexity-user.json

# Safeguard: an abnormally short list signals a download problem
[ "$(wc -l < "$tmp")" -ge 10 ] || { echo "list too short, aborting" >&2; exit 1; }

sort -u "$tmp" > "$output.new"
cp -p "$output" "$output.bak" 2>/dev/null || true
mv "$output.new" "$output"

if ! nginx -t; then
  echo "configuration rejected, restoring the previous list" >&2
  [ -f "$output.bak" ] && mv "$output.bak" "$output"
  exit 1
fi
systemctl reload nginx
```

```text
# /etc/cron.d/update-ai-crawlers: every day at 4:17 a.m.
17 4 * * * root /usr/local/sbin/update-ai-crawlers-nginx.sh
```

With `set -e`, a failed download stops the script and the previous list stays in place. Before the first run, create an empty file (`touch /etc/nginx/ai-crawlers-ip.conf`) so that `nginx -t` succeeds. To check by hand an address you spotted in your logs, the [IABIG crawler IP checker](/tools/verify-bot-ip/) queries the same lists. If you would rather not maintain the user-agent list yourself, the [AI blocklists page](/tools/ai-blocklists/) publishes ready-made nginx rules (as well as robots.txt, Apache and Cloudflare versions), regenerated every hour from our bot database.

## X-Robots-Tag, if needed

```nginx
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}
```

Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. Watch out for inheritance: `add_header` directives from one level are only inherited if the current level declares none of its own. An `add_header` in this `location` makes those of the `server` block disappear.

## Testing

```bash
nginx -t && systemctl reload nginx

# Browser: 200
curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/
# Training crawler: 403, but 200 on robots.txt
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Search crawler announced from your own machine: 403 (IP outside the ranges)
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; OAI-SearchBot/1.4" https://www.example.com/

tail -f /var/log/nginx/ai-crawlers.log
```

These commands are only indicative: curl reproduces the User-Agent, not the crawler's address. The last test shows this well: from your own machine, a fake `OAI-SearchBot` is denied, which is the expected behavior. In our tests, after declaring the local address as an OpenAI range, `OAI-SearchBot` got a 200, then a 429 once the burst was exceeded; `GPTBot` received a 403 on `/` and a 200 on `/robots.txt`. The real check then happens in `ai-crawlers.log`, where the `ip=` column shows the vendor recognized for each request.

## Pitfalls to avoid

- **The User-Agent alone proves nothing.** Denying by name stops honest crawlers, which robots.txt already stopped; impostors are only caught by comparing against IP ranges, and only for vendors that publish them.
- **An outdated list blocks genuine crawlers.** Ranges change; keep an eye on when the files were generated and on `denied=1` log lines involving search crawlers.
- **An overly broad expression denies everyone.** `~*bot` would catch Googlebot and Bingbot.
- **Blocking a vendor's IP ranges to refuse training.** Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.

## Rolling back

Comment out the `if ($ai_deny) { return 403; }` line and the `limit_req` directive, then run `nginx -t` and `systemctl reload nginx`: traffic is no longer filtered but is still logged. To remove everything, delete `/etc/nginx/conf.d/ai-crawlers.conf`, the lines added to the `server` block and the cron job.

## Official sources

- nginx, map module: <https://nginx.org/en/docs/http/ngx_http_map_module.html>
- nginx, geo module: <https://nginx.org/en/docs/http/ngx_http_geo_module.html>
- nginx, limit_req module: <https://nginx.org/en/docs/http/ngx_http_limit_req_module.html>
- nginx, log module (access_log, log_format): <https://nginx.org/en/docs/http/ngx_http_log_module.html>
- nginx, headers module (add_header): <https://nginx.org/en/docs/http/ngx_http_headers_module.html>
- nginx, realip module: <https://nginx.org/en/docs/http/ngx_http_realip_module.html>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Google, Robots meta tag and X-Robots-Tag: <https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag>
