IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNS

Confirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.

Any script can send a User-Agent: GPTBot header. In a site's logs, some of the visits attributed to AI crawlers actually come from scrapers that borrow the name to get past filters, or simply to blend into the traffic. Common Crawl even warns about it on its own page: some bots pretend to be CCBot. The only way to settle the question is to compare the request's IP address with what the operator publishes. This guide gives the documented method for each crawler, ready-to-use commands, and then shows how to block impostors with nginx and fail2ban.

Which method for which crawler

Operators do not all use the same method. Most publish a JSON file of IP ranges. Google, Bing, Apple and Common Crawl also document verification by reverse DNS followed by a forward lookup.

Crawler Documented method Source of truth
GPTBot IP ranges https://openai.com/gptbot.json
OAI-SearchBot IP ranges https://openai.com/searchbot.json
ChatGPT-User IP ranges https://openai.com/chatgpt-user.json
ClaudeBot, Claude-SearchBot, Claude-User IP ranges (one list for all Anthropic crawlers) https://claude.com/crawling/bots.json
PerplexityBot IP ranges https://www.perplexity.ai/perplexitybot.json
Perplexity-User IP ranges https://www.perplexity.ai/perplexity-user.json
Googlebot and Google fetchers Reverse + forward DNS, or IP ranges googlebot.com, google.com, googleusercontent.com
Applebot Reverse DNS or IP ranges *.applebot.apple.com, https://search.developer.apple.com/applebot.json
CCBot IP ranges, reverse + forward DNS over IPv4 *.crawl.commoncrawl.org, https://index.commoncrawl.org/ccbot.json
Bingbot (also feeds Microsoft Copilot) Reverse + forward DNS, or IP ranges *.search.msn.com, https://www.bing.com/toolbox/bingbot.json
DuckAssistBot IP ranges https://duckduckgo.com/duckassistbot.json
MistralAI-User IP ranges https://mistral.ai/mistralai-user-ips.json

OpenAI, Anthropic, Perplexity, DuckDuckGo and Mistral do not document a reverse DNS hostname. For these crawlers, do not base a decision on DNS; use the JSON files. They share the same layout (the one Google and Bing also use): a top-level creationTime timestamp and a prefixes array in which each item holds an ipv4Prefix or ipv6Prefix in CIDR notation. Some files add extra fields, such as Common Crawl's synctoken and notes, which you can ignore.

{
  "creationTime": "...",
  "prefixes": [
    { "ipv4Prefix": "192.0.2.0/24" },
    { "ipv6Prefix": "2001:db8::/32" }
  ]
}

The ranges change, and some files are regenerated very often (the creationTime of OpenAI's chatgpt-user.json is frequently only hours old). Download them regularly instead of copying them once and for all.

Extracting the ranges with curl and jq

# CIDR prefixes published for GPTBot
curl -fsSL https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'

# File generation date
curl -fsSL https://claude.com/crawling/bots.json | jq -r '.creationTime'

# Number of prefixes per file
for u in https://openai.com/gptbot.json \
         https://openai.com/searchbot.json \
         https://claude.com/crawling/bots.json \
         https://www.perplexity.ai/perplexitybot.json; do
  printf '%5d  %s\n' "$(curl -fsSL "$u" | jq '.prefixes | length')" "$u"
done

jq cannot test whether an address belongs to a CIDR network. For that step, Python and its standard ipaddress module are more reliable than a regular expression.

Testing an IP address with Python

The following script downloads the official lists and tells you which crawler an address belongs to. It only uses the standard library.

#!/usr/bin/env python3
"""Tells whether an IP address belongs to the published ranges of an AI crawler."""
import ipaddress
import json
import sys
import urllib.request

SOURCES = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "Anthropic (ClaudeBot, Claude-SearchBot, Claude-User)": "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}


def prefixes(url):
    req = urllib.request.Request(url, headers={"User-Agent": "verify-ip/1.0"})
    with urllib.request.urlopen(req, timeout=15) as resp:
        data = json.load(resp)
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            yield ipaddress.ip_network(cidr, strict=False)


def main():
    if len(sys.argv) != 2:
        sys.exit("usage: verify_bot.py <IP address>")
    ip = ipaddress.ip_address(sys.argv[1])
    found = False
    for name, url in SOURCES.items():
        try:
            networks = list(prefixes(url))
        except Exception as exc:  # network error, invalid JSON...
            print(f"[error] {name}: {exc}", file=sys.stderr)
            continue
        for net in networks:
            if ip.version == net.version and ip in net:
                print(f"{ip} belongs to {name} ({net})")
                found = True
    if not found:
        print(f"{ip} is not in any of the published ranges checked")
    sys.exit(0 if found else 1)


if __name__ == "__main__":
    main()
python3 verify_bot.py 20.171.206.5
# 20.171.206.5 belongs to GPTBot (20.171.206.0/24)   (result as of November 6, 2026)

# Check every IP that claims to be GPTBot in a log file
grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort -u |
  while read -r ip; do python3 verify_bot.py "$ip" >/dev/null || echo "SUSPECT $ip"; done

The exit code (0 if the address is found, 1 otherwise) makes it easy to use in a script. For large volumes, load the lists once rather than rerunning the script for each address.

Mind the granularity: Anthropic's list covers all its crawlers without telling them apart. An IP on that list proves the request comes from Anthropic, not that it is ClaudeBot rather than Claude-User.

Reverse then forward DNS, where that is the documented method

For Google, Bing, Apple and Common Crawl, verification takes two steps: a reverse lookup of the IP to a hostname, a check of the domain, then a forward lookup of that name, which must return the same IP. The second step is essential, because whoever owns an IP address can set its PTR record to anything.

# Example published by Google
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# Apple: the name must end in .applebot.apple.com
host 17.58.101.179

# Common Crawl: the name must end in .crawl.commoncrawl.org (IPv4 only)
dig +short -x 18.97.14.84
dig +short 18-97-14-84.crawl.commoncrawl.org A

The same logic in Python, for automation:

#!/usr/bin/env python3
"""Reverse then forward DNS verification (FCrDNS)."""
import ipaddress
import socket
import sys

DOMAINS = {
    "googlebot": (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "applebot": (".applebot.apple.com",),
    "ccbot": (".crawl.commoncrawl.org",),
    "bingbot": (".search.msn.com",),
}


def fcrdns(ip, suffixes):
    try:
        name = socket.gethostbyaddr(ip)[0].rstrip(".")
    except OSError:
        return None, False
    if not name.endswith(suffixes):
        return name, False
    try:
        addresses = {ipaddress.ip_address(ai[4][0].split("%")[0])
                    for ai in socket.getaddrinfo(name, None)}
    except OSError:
        return name, False
    return name, ipaddress.ip_address(ip) in addresses


if __name__ == "__main__":
    if len(sys.argv) != 3 or sys.argv[1] not in DOMAINS:
        sys.exit("usage: fcrdns.py googlebot|applebot|ccbot|bingbot <IP address>")
    name, ok = fcrdns(sys.argv[2], DOMAINS[sys.argv[1]])
    print(f"{sys.argv[2]} -> {name}: {'authentic' if ok else 'NOT verified'}")
    sys.exit(0 if ok else 1)

The check is on the suffix preceded by a dot, so googlebot.com.example.net or fake-googlebot.com are rejected. Google notes that user-triggered fetchers resolve to google.com or gae.googleusercontent.com depending on the case. A googleusercontent.com name can therefore belong to a service hosted on Google Cloud rather than to Google itself.

Blocking impostors with nginx

The most robust approach is to let nginx compare, for every request, the crawler being claimed with the operator that owns the IP. First, generate a range file for the geo module:

#!/usr/bin/env bash
# /usr/local/sbin/update-ai-bot-ips.sh - run daily (cron or systemd timer)
set -euo pipefail
output=/etc/nginx/ai-bot-ips.conf
tmp=$(mktemp)
add() {  # $1 = operator, $2 = URL of the official JSON
  curl -fsSL "$2" | jq -r --arg v "$1" '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " " + $v + ";"' >> "$tmp"
}
add openai     https://openai.com/gptbot.json
add openai     https://openai.com/searchbot.json
add openai     https://openai.com/chatgpt-user.json
add anthropic  https://claude.com/crawling/bots.json
add perplexity https://www.perplexity.ai/perplexitybot.json
add perplexity https://www.perplexity.ai/perplexity-user.json
[ -s "$tmp" ] || { echo "empty list, aborting" >&2; exit 1; }
sort -u "$tmp" > "$output.new" && rm -f "$tmp"
mv "$output.new" "$output"
nginx -t && systemctl reload nginx

With set -e, a failed download stops the script and the previous list stays in place. The nginx configuration, in the http block:

# Operator that owns the IP according to the official lists
geo $ip_ai_operator {
    default none;
    include /etc/nginx/ai-bot-ips.conf;
}

# Operator claimed by the User-Agent
map $http_user_agent $ua_ai_operator {
    default                                   "";
    "~*GPTBot|OAI-SearchBot|ChatGPT-User"     openai;
    "~*ClaudeBot|Claude-SearchBot|Claude-User" anthropic;
    "~*PerplexityBot|Perplexity-User"         perplexity;
}

# 1 = crawler claimed but IP outside its operator's ranges
map "$ua_ai_operator:$ip_ai_operator" $spoofed_bot {
    default                1;
    "~^:"                  0;   # no AI crawler claimed
    "openai:openai"        0;
    "anthropic:anthropic"  0;
    "perplexity:perplexity" 0;
}

log_format spoofed_bots '$remote_addr [$time_local] "$request" "$http_user_agent"';

Then in the site's server block:

# An access_log directive in server replaces those inherited from http,
# so the main log is declared again.
access_log /var/log/nginx/access.log combined;
access_log /var/log/nginx/spoofed-bots.log spoofed_bots if=$spoofed_bot;
if ($spoofed_bot) {
    return 403;
}

Genuine crawlers get through, impostors receive a 403 and are recorded in a dedicated log. To block a specific address right away, the standard directive is enough: deny 203.0.113.7; in the server block, or an included file that groups these lines.

If your site sits behind a CDN or a proxy, $remote_addr holds the proxy's address. Configure the realip module first (set_real_ip_from and real_ip_header) to recover the client IP, otherwise genuine crawlers will be classified as impostors.

Banning repeat offenders with fail2ban

The spoofed-bots.log file only contains requests already identified as fraudulent, so fail2ban can use it to ban the IP at the firewall level, which also protects the machine's other services.

# /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
[Definition]
failregex = ^<HOST> \[
ignoreregex =
# /etc/fail2ban/jail.d/nginx-spoofed-bots.local
[nginx-spoofed-bots]
enabled  = true
filter   = nginx-spoofed-bots
logpath  = /var/log/nginx/spoofed-bots.log
port     = http,https
maxretry = 3
findtime = 10m
bantime  = 1d
fail2ban-regex /var/log/nginx/spoofed-bots.log /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
systemctl reload fail2ban
fail2ban-client status nginx-spoofed-bots

Behind a CDN, a ban at the machine's firewall does nothing, since connections arrive from the CDN. Apply the rule at the CDN level instead.

Choosing a proportionate response

An impostor is not necessarily malicious. It may be a testing tool, a monitoring service or a developer checking their own robots.txt. Scale the response to the volume and behavior you observe.

Situation Proportionate response
A few isolated requests Log them, do not block
Sustained crawling with a fake User-Agent 403 via nginx, as above
Aggressive crawling from several addresses fail2ban ban, or a rule at the CDN level
Excessive load from a genuine crawler robots.txt (and Crawl-delay for crawlers that honor it, such as ClaudeBot and CCBot), then contact the operator

For a genuine crawler that is too aggressive, IP blocking is not the right path. Anthropic, Amazon (amazonbot@amazon.com) and Meta (webmasters@meta.com) publish a contact address in their documentation. Anthropic asks you to write from an address on the affected domain, and Amazon asks you to include the domain names concerned.

Finally, watch the impostor log after each list update. A sudden rise in "genuine" crawlers classified as impostors almost always points to a stale list or a CDN problem, not an attack.

What not to do

  • Block an operator's IP ranges to opt out of training. Anthropic says this method is not reliable, because it also prevents the crawler from reading robots.txt. Use robots.txt to express an opt-out.
  • Rely on reverse DNS for OpenAI, Anthropic, Perplexity, DuckDuckGo or Mistral. None of their documentation provides for it.
  • Copy the ranges once and for all. They change, and a stale list will eventually block a genuine crawler.
  • Deploy without testing. An overly broad regular expression in map can classify legitimate traffic as an impostor. Check the configuration with nginx -t, the filter with fail2ban-regex, and read the first lines of spoofed-bots.log.

To check a single address against the published ranges without running a script, use the crawler IP verification tool. The operators' tokens and IP files are listed in the AI crawler user agents list.

Official sources

Free tools

Related guides