IABIGFR · Français

Platform guides · Updated 2026-11-06 · Markdown version

How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate Limits

A tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.

With nginx, everything is handled in the server configuration: you can serve robots.txt, deny a request based on its User-Agent, check that a crawler really comes from its vendor's addresses, limit its request rate and write it to a separate log. This guide assembles those pieces into a complete configuration built around a simple policy: refuse model training, let the search crawlers of ChatGPT, Claude and Perplexity through, and return a 403 to anything that borrows their names without coming from their IP ranges. We validated this configuration with nginx -t and then with real requests on nginx 1.30.6, using the official Windows build; the Linux paths shown (/etc/nginx, /var/log/nginx) are those of the Debian and Ubuntu packages and should be adjusted for your distribution.

robots.txt first

Server-side blocking does not replace robots.txt: it is the file that crawlers which respect your choices actually read, and the only way to refuse training to Google (Google-Extended) or Apple (Applebot-Extended), which never send requests under those names. Put the file at the site root and make sure it always returns a 200, whatever the crawler:

location = /robots.txt {
    # Served to everyone, including crawlers denied elsewhere
    try_files $uri =404;
    access_log off;   # optional: keeps the main log uncluttered
}

RFC 9309 specifies that a robots.txt that is unreachable because of a 5xx error counts as a full disallow, and a robots.txt denied with a 403 prevents the crawler from reading your refusal in it. The configuration below therefore exempts /robots.txt from every block.

The http block configuration

This file goes in the http context, for example in /etc/nginx/conf.d/, which Debian and Ubuntu include by default. Each map costs nothing until its variable is used, since nginx evaluates it on demand.

# /etc/nginx/conf.d/ai-crawlers.conf  (http context)

# 1. AI crawler announced by the User-Agent.
#    Expressions are tested in order; ~* = case-insensitive.
map $http_user_agent $ai_crawler {
    default                 "";
    "~*GPTBot"              GPTBot;
    "~*OAI-SearchBot"       OAI-SearchBot;
    "~*ChatGPT-User"        ChatGPT-User;
    "~*ClaudeBot"           ClaudeBot;
    "~*Claude-SearchBot"    Claude-SearchBot;
    "~*Claude-User"         Claude-User;
    "~*PerplexityBot"       PerplexityBot;
    "~*Perplexity-User"     Perplexity-User;
    "~*CCBot"               CCBot;
    "~*meta-externalagent"  meta-externalagent;
    "~*Bytespider"          Bytespider;
}

# 2. Crawlers denied server-side (here: training; adjust as needed)
map $ai_crawler $crawler_denied {
    default             0;
    GPTBot              1;
    ClaudeBot           1;
    CCBot               1;
    meta-externalagent  1;
    Bytespider          1;
}

# 3. Claimed vendor, for crawlers whose IP ranges are published
map $ai_crawler $claimed_vendor {
    default           "";
    GPTBot            openai;
    OAI-SearchBot     openai;
    ChatGPT-User      openai;
    ClaudeBot         anthropic;
    Claude-SearchBot  anthropic;
    Claude-User       anthropic;
    PerplexityBot     perplexity;
    Perplexity-User   perplexity;
}

# 4. Actual vendor based on the IP address (file generated by the script below)
geo $ip_vendor {
    default none;
    include /etc/nginx/ai-crawlers-ip.conf;
}

# 5. 1 = announced crawler, but IP outside its vendor's ranges
map "$claimed_vendor:$ip_vendor" $crawler_spoofed {
    default                  1;
    "~^:"                    0;   # no verifiable crawler announced
    "openai:openai"          0;
    "anthropic:anthropic"    0;
    "perplexity:perplexity"  0;
}

# 6. Final decision. robots.txt always stays readable.
map "$crawler_denied$crawler_spoofed:$uri" $ai_deny {
    default                  0;
    "~^..:/robots\.txt$"     0;   # must remain the first expression
    "~^1"                    1;   # crawler denied on principle
    "~^01"                   1;   # spoofed crawler
}

# 7. Rate limiting: only AI crawlers get a key.
#    An empty key is not counted.
map $ai_crawler $ai_limit_key {
    ""       "";
    default  $binary_remote_addr;
}
limit_req_zone $ai_limit_key zone=ai_crawlers:10m rate=1r/s;

# 8. Format of the dedicated log
log_format ai_crawlers '$remote_addr [$time_local] "$request" $status '
                     'crawler=$ai_crawler ip=$ip_vendor denied=$ai_deny "$http_user_agent"';

A few notes on reading it. In a map, the nginx documentation states that the search stops at the first match, with exact strings checked before regular expressions, which are themselves tested in the order they appear in the file. The pattern ~*GPTBot does not match ChatGPT-User (which contains "GPT-U", not "GPTBot"), and ~*ClaudeBot does not match Claude-SearchBot, so the order above is unambiguous. The geo module accepts IPv4 and IPv6 prefixes in CIDR notation. Finally, limit_req_zone ignores requests whose key is empty, so ordinary visitors are not rate-limited.

The server block

server {
    listen 443 ssl;
    server_name www.example.com;
    # ssl_certificate and ssl_certificate_key: unchanged, omitted here
    root /var/www/example;

    # An access_log declared here replaces those inherited from http:
    # redeclare the main log, then the dedicated log.
    access_log /var/log/nginx/access.log combined;
    access_log /var/log/nginx/ai-crawlers.log ai_crawlers if=$ai_crawler;

    # 1 request/s per address for AI crawlers, bursts of 5 tolerated
    limit_req zone=ai_crawlers burst=5 nodelay;
    limit_req_status 429;

    # Deny (training crawlers and impostors), except for robots.txt
    if ($ai_deny) {
        return 403;
    }

    location = /robots.txt {
        try_files $uri =404;
    }

    location / {
        try_files $uri $uri/ =404;
    }
}

The rewrite module's if and return directives are allowed in the server context; placed there, they apply to every URL, whichever location is selected afterwards. The if= parameter of access_log only writes the line when the variable is neither empty nor equal to "0", so ai-crawlers.log only contains requests from an announced AI crawler, whether accepted or denied. The 429 status set by limit_req_status (503 by default) clearly tells the crawler to slow down.

If the site sits behind a CDN or a load balancer, $remote_addr holds that intermediary's address: configure the realip module first (set_real_ip_from, real_ip_header), otherwise every genuine crawler will be classified as an impostor.

Keeping the official IP ranges up to date

OpenAI, Anthropic and Perplexity publish their ranges in JSON files with the same format: a prefixes array in which each element contains ipv4Prefix or ipv6Prefix. The following script, which we ran against the live files in October 2026 (a little over 300 prefixes in total), produces the file included by geo and only reloads nginx if the configuration is still valid.

#!/usr/bin/env bash
# /usr/local/sbin/update-ai-crawlers-nginx.sh - requires curl and jq
set -euo pipefail
output=/etc/nginx/ai-crawlers-ip.conf
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

add() {  # $1 = vendor, $2 = URL of the official JSON file
  # -L: follow redirects if a list moves
  curl -fsSL --max-time 30 "$2" \
    | jq -er --arg e "$1" '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null) | "\(.) \($e);"' >> "$tmp"
}
add openai     https://openai.com/gptbot.json
add openai     https://openai.com/searchbot.json
add openai     https://openai.com/chatgpt-user.json
add anthropic  https://claude.com/crawling/bots.json
add perplexity https://www.perplexity.ai/perplexitybot.json
add perplexity https://www.perplexity.ai/perplexity-user.json

# Safeguard: an abnormally short list signals a download problem
[ "$(wc -l < "$tmp")" -ge 10 ] || { echo "list too short, aborting" >&2; exit 1; }

sort -u "$tmp" > "$output.new"
cp -p "$output" "$output.bak" 2>/dev/null || true
mv "$output.new" "$output"

if ! nginx -t; then
  echo "configuration rejected, restoring the previous list" >&2
  [ -f "$output.bak" ] && mv "$output.bak" "$output"
  exit 1
fi
systemctl reload nginx
# /etc/cron.d/update-ai-crawlers: every day at 4:17 a.m.
17 4 * * * root /usr/local/sbin/update-ai-crawlers-nginx.sh

With set -e, a failed download stops the script and the previous list stays in place. Before the first run, create an empty file (touch /etc/nginx/ai-crawlers-ip.conf) so that nginx -t succeeds. To check by hand an address you spotted in your logs, the IABIG crawler IP checker queries the same lists. If you would rather not maintain the user-agent list yourself, the AI blocklists page publishes ready-made nginx rules (as well as robots.txt, Apache and Cloudflare versions), regenerated every hour from our bot database.

X-Robots-Tag, if needed

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. Watch out for inheritance: add_header directives from one level are only inherited if the current level declares none of its own. An add_header in this location makes those of the server block disappear.

Testing

nginx -t && systemctl reload nginx

# Browser: 200
curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/
# Training crawler: 403, but 200 on robots.txt
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Search crawler announced from your own machine: 403 (IP outside the ranges)
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; OAI-SearchBot/1.4" https://www.example.com/

tail -f /var/log/nginx/ai-crawlers.log

These commands are only indicative: curl reproduces the User-Agent, not the crawler's address. The last test shows this well: from your own machine, a fake OAI-SearchBot is denied, which is the expected behavior. In our tests, after declaring the local address as an OpenAI range, OAI-SearchBot got a 200, then a 429 once the burst was exceeded; GPTBot received a 403 on / and a 200 on /robots.txt. The real check then happens in ai-crawlers.log, where the ip= column shows the vendor recognized for each request.

Pitfalls to avoid

  • The User-Agent alone proves nothing. Denying by name stops honest crawlers, which robots.txt already stopped; impostors are only caught by comparing against IP ranges, and only for vendors that publish them.
  • An outdated list blocks genuine crawlers. Ranges change; keep an eye on when the files were generated and on denied=1 log lines involving search crawlers.
  • An overly broad expression denies everyone. ~*bot would catch Googlebot and Bingbot.
  • Blocking a vendor's IP ranges to refuse training. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.

Rolling back

Comment out the if ($ai_deny) { return 403; } line and the limit_req directive, then run nginx -t and systemctl reload nginx: traffic is no longer filtered but is still logged. To remove everything, delete /etc/nginx/conf.d/ai-crawlers.conf, the lines added to the server block and the cron job.

Official sources

Free tools

Related guides