# How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNS

> Confirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.

Source: https://iabig.com/guides/ai-crawlers/verify-gptbot-claudebot-perplexitybot/ — Guides CC BY 4.0, extraits de code MIT

Any script can send a `User-Agent: GPTBot` header. In a site's logs, some of the visits attributed to AI crawlers actually come from scrapers that borrow the name to get past filters, or simply to blend into the traffic. Common Crawl even warns about it on its own page: some bots pretend to be CCBot. The only way to settle the question is to compare the request's IP address with what the operator publishes. This guide gives the documented method for each crawler, ready-to-use commands, and then shows how to block impostors with nginx and fail2ban.

## Which method for which crawler

Operators do not all use the same method. Most publish a JSON file of IP ranges. Google, Bing, Apple and Common Crawl also document verification by reverse DNS followed by a forward lookup.

| Crawler | Documented method | Source of truth |
|---|---|---|
| `GPTBot` | IP ranges | `https://openai.com/gptbot.json` |
| `OAI-SearchBot` | IP ranges | `https://openai.com/searchbot.json` |
| `ChatGPT-User` | IP ranges | `https://openai.com/chatgpt-user.json` |
| `ClaudeBot`, `Claude-SearchBot`, `Claude-User` | IP ranges (one list for all Anthropic crawlers) | `https://claude.com/crawling/bots.json` |
| `PerplexityBot` | IP ranges | `https://www.perplexity.ai/perplexitybot.json` |
| `Perplexity-User` | IP ranges | `https://www.perplexity.ai/perplexity-user.json` |
| `Googlebot` and Google fetchers | Reverse + forward DNS, or IP ranges | `googlebot.com`, `google.com`, `googleusercontent.com` |
| `Applebot` | Reverse DNS or IP ranges | `*.applebot.apple.com`, `https://search.developer.apple.com/applebot.json` |
| `CCBot` | IP ranges, reverse + forward DNS over IPv4 | `*.crawl.commoncrawl.org`, `https://index.commoncrawl.org/ccbot.json` |
| `Bingbot` (also feeds Microsoft Copilot) | Reverse + forward DNS, or IP ranges | `*.search.msn.com`, `https://www.bing.com/toolbox/bingbot.json` |
| `DuckAssistBot` | IP ranges | `https://duckduckgo.com/duckassistbot.json` |
| `MistralAI-User` | IP ranges | `https://mistral.ai/mistralai-user-ips.json` |

OpenAI, Anthropic, Perplexity, DuckDuckGo and Mistral do not document a reverse DNS hostname. For these crawlers, do not base a decision on DNS; use the JSON files. They share the same layout (the one Google and Bing also use): a top-level `creationTime` timestamp and a `prefixes` array in which each item holds an `ipv4Prefix` or `ipv6Prefix` in CIDR notation. Some files add extra fields, such as Common Crawl's `synctoken` and `notes`, which you can ignore.

```text
{
  "creationTime": "...",
  "prefixes": [
    { "ipv4Prefix": "192.0.2.0/24" },
    { "ipv6Prefix": "2001:db8::/32" }
  ]
}
```

The ranges change, and some files are regenerated very often (the `creationTime` of OpenAI's `chatgpt-user.json` is frequently only hours old). Download them regularly instead of copying them once and for all.

## Extracting the ranges with curl and jq

```bash
# CIDR prefixes published for GPTBot
curl -fsSL https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'

# File generation date
curl -fsSL https://claude.com/crawling/bots.json | jq -r '.creationTime'

# Number of prefixes per file
for u in https://openai.com/gptbot.json \
         https://openai.com/searchbot.json \
         https://claude.com/crawling/bots.json \
         https://www.perplexity.ai/perplexitybot.json; do
  printf '%5d  %s\n' "$(curl -fsSL "$u" | jq '.prefixes | length')" "$u"
done
```

jq cannot test whether an address belongs to a CIDR network. For that step, Python and its standard `ipaddress` module are more reliable than a regular expression.

## Testing an IP address with Python

The following script downloads the official lists and tells you which crawler an address belongs to. It only uses the standard library.

```python
#!/usr/bin/env python3
"""Tells whether an IP address belongs to the published ranges of an AI crawler."""
import ipaddress
import json
import sys
import urllib.request

SOURCES = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "Anthropic (ClaudeBot, Claude-SearchBot, Claude-User)": "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}


def prefixes(url):
    req = urllib.request.Request(url, headers={"User-Agent": "verify-ip/1.0"})
    with urllib.request.urlopen(req, timeout=15) as resp:
        data = json.load(resp)
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            yield ipaddress.ip_network(cidr, strict=False)


def main():
    if len(sys.argv) != 2:
        sys.exit("usage: verify_bot.py <IP address>")
    ip = ipaddress.ip_address(sys.argv[1])
    found = False
    for name, url in SOURCES.items():
        try:
            networks = list(prefixes(url))
        except Exception as exc:  # network error, invalid JSON...
            print(f"[error] {name}: {exc}", file=sys.stderr)
            continue
        for net in networks:
            if ip.version == net.version and ip in net:
                print(f"{ip} belongs to {name} ({net})")
                found = True
    if not found:
        print(f"{ip} is not in any of the published ranges checked")
    sys.exit(0 if found else 1)


if __name__ == "__main__":
    main()
```

```bash
python3 verify_bot.py 20.171.206.5
# 20.171.206.5 belongs to GPTBot (20.171.206.0/24)   (result as of November 6, 2026)

# Check every IP that claims to be GPTBot in a log file
grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort -u |
  while read -r ip; do python3 verify_bot.py "$ip" >/dev/null || echo "SUSPECT $ip"; done
```

The exit code (0 if the address is found, 1 otherwise) makes it easy to use in a script. For large volumes, load the lists once rather than rerunning the script for each address.

Mind the granularity: Anthropic's list covers all its crawlers without telling them apart. An IP on that list proves the request comes from Anthropic, not that it is ClaudeBot rather than Claude-User.

## Reverse then forward DNS, where that is the documented method

For Google, Bing, Apple and Common Crawl, verification takes two steps: a reverse lookup of the IP to a hostname, a check of the domain, then a forward lookup of that name, which must return the same IP. The second step is essential, because whoever owns an IP address can set its PTR record to anything.

```bash
# Example published by Google
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# Apple: the name must end in .applebot.apple.com
host 17.58.101.179

# Common Crawl: the name must end in .crawl.commoncrawl.org (IPv4 only)
dig +short -x 18.97.14.84
dig +short 18-97-14-84.crawl.commoncrawl.org A
```

The same logic in Python, for automation:

```python
#!/usr/bin/env python3
"""Reverse then forward DNS verification (FCrDNS)."""
import ipaddress
import socket
import sys

DOMAINS = {
    "googlebot": (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "applebot": (".applebot.apple.com",),
    "ccbot": (".crawl.commoncrawl.org",),
    "bingbot": (".search.msn.com",),
}


def fcrdns(ip, suffixes):
    try:
        name = socket.gethostbyaddr(ip)[0].rstrip(".")
    except OSError:
        return None, False
    if not name.endswith(suffixes):
        return name, False
    try:
        addresses = {ipaddress.ip_address(ai[4][0].split("%")[0])
                    for ai in socket.getaddrinfo(name, None)}
    except OSError:
        return name, False
    return name, ipaddress.ip_address(ip) in addresses


if __name__ == "__main__":
    if len(sys.argv) != 3 or sys.argv[1] not in DOMAINS:
        sys.exit("usage: fcrdns.py googlebot|applebot|ccbot|bingbot <IP address>")
    name, ok = fcrdns(sys.argv[2], DOMAINS[sys.argv[1]])
    print(f"{sys.argv[2]} -> {name}: {'authentic' if ok else 'NOT verified'}")
    sys.exit(0 if ok else 1)
```

The check is on the suffix preceded by a dot, so `googlebot.com.example.net` or `fake-googlebot.com` are rejected. Google notes that user-triggered fetchers resolve to `google.com` or `gae.googleusercontent.com` depending on the case. A `googleusercontent.com` name can therefore belong to a service hosted on Google Cloud rather than to Google itself.

## Blocking impostors with nginx

The most robust approach is to let nginx compare, for every request, the crawler being claimed with the operator that owns the IP. First, generate a range file for the `geo` module:

```bash
#!/usr/bin/env bash
# /usr/local/sbin/update-ai-bot-ips.sh - run daily (cron or systemd timer)
set -euo pipefail
output=/etc/nginx/ai-bot-ips.conf
tmp=$(mktemp)
add() {  # $1 = operator, $2 = URL of the official JSON
  curl -fsSL "$2" | jq -r --arg v "$1" '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " " + $v + ";"' >> "$tmp"
}
add openai     https://openai.com/gptbot.json
add openai     https://openai.com/searchbot.json
add openai     https://openai.com/chatgpt-user.json
add anthropic  https://claude.com/crawling/bots.json
add perplexity https://www.perplexity.ai/perplexitybot.json
add perplexity https://www.perplexity.ai/perplexity-user.json
[ -s "$tmp" ] || { echo "empty list, aborting" >&2; exit 1; }
sort -u "$tmp" > "$output.new" && rm -f "$tmp"
mv "$output.new" "$output"
nginx -t && systemctl reload nginx
```

With `set -e`, a failed download stops the script and the previous list stays in place. The nginx configuration, in the `http` block:

```nginx
# Operator that owns the IP according to the official lists
geo $ip_ai_operator {
    default none;
    include /etc/nginx/ai-bot-ips.conf;
}

# Operator claimed by the User-Agent
map $http_user_agent $ua_ai_operator {
    default                                   "";
    "~*GPTBot|OAI-SearchBot|ChatGPT-User"     openai;
    "~*ClaudeBot|Claude-SearchBot|Claude-User" anthropic;
    "~*PerplexityBot|Perplexity-User"         perplexity;
}

# 1 = crawler claimed but IP outside its operator's ranges
map "$ua_ai_operator:$ip_ai_operator" $spoofed_bot {
    default                1;
    "~^:"                  0;   # no AI crawler claimed
    "openai:openai"        0;
    "anthropic:anthropic"  0;
    "perplexity:perplexity" 0;
}

log_format spoofed_bots '$remote_addr [$time_local] "$request" "$http_user_agent"';
```

Then in the site's `server` block:

```nginx
# An access_log directive in server replaces those inherited from http,
# so the main log is declared again.
access_log /var/log/nginx/access.log combined;
access_log /var/log/nginx/spoofed-bots.log spoofed_bots if=$spoofed_bot;
if ($spoofed_bot) {
    return 403;
}
```

Genuine crawlers get through, impostors receive a 403 and are recorded in a dedicated log. To block a specific address right away, the standard directive is enough: `deny 203.0.113.7;` in the `server` block, or an included file that groups these lines.

If your site sits behind a CDN or a proxy, `$remote_addr` holds the proxy's address. Configure the `realip` module first (`set_real_ip_from` and `real_ip_header`) to recover the client IP, otherwise genuine crawlers will be classified as impostors.

## Banning repeat offenders with fail2ban

The `spoofed-bots.log` file only contains requests already identified as fraudulent, so fail2ban can use it to ban the IP at the firewall level, which also protects the machine's other services.

```text
# /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
[Definition]
failregex = ^<HOST> \[
ignoreregex =
```

```text
# /etc/fail2ban/jail.d/nginx-spoofed-bots.local
[nginx-spoofed-bots]
enabled  = true
filter   = nginx-spoofed-bots
logpath  = /var/log/nginx/spoofed-bots.log
port     = http,https
maxretry = 3
findtime = 10m
bantime  = 1d
```

```bash
fail2ban-regex /var/log/nginx/spoofed-bots.log /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
systemctl reload fail2ban
fail2ban-client status nginx-spoofed-bots
```

Behind a CDN, a ban at the machine's firewall does nothing, since connections arrive from the CDN. Apply the rule at the CDN level instead.

## Choosing a proportionate response

An impostor is not necessarily malicious. It may be a testing tool, a monitoring service or a developer checking their own robots.txt. Scale the response to the volume and behavior you observe.

| Situation | Proportionate response |
|---|---|
| A few isolated requests | Log them, do not block |
| Sustained crawling with a fake User-Agent | 403 via nginx, as above |
| Aggressive crawling from several addresses | fail2ban ban, or a rule at the CDN level |
| Excessive load from a genuine crawler | robots.txt (and `Crawl-delay` for crawlers that honor it, such as ClaudeBot and CCBot), then contact the operator |

For a genuine crawler that is too aggressive, IP blocking is not the right path. Anthropic, Amazon (`amazonbot@amazon.com`) and Meta (`webmasters@meta.com`) publish a contact address in their documentation. Anthropic asks you to write from an address on the affected domain, and Amazon asks you to include the domain names concerned.

Finally, watch the impostor log after each list update. A sudden rise in "genuine" crawlers classified as impostors almost always points to a stale list or a CDN problem, not an attack.

## What not to do

- Block an operator's IP ranges to opt out of training. Anthropic says this method is not reliable, because it also prevents the crawler from reading robots.txt. Use robots.txt to express an opt-out.
- Rely on reverse DNS for OpenAI, Anthropic, Perplexity, DuckDuckGo or Mistral. None of their documentation provides for it.
- Copy the ranges once and for all. They change, and a stale list will eventually block a genuine crawler.
- Deploy without testing. An overly broad regular expression in `map` can classify legitimate traffic as an impostor. Check the configuration with `nginx -t`, the filter with `fail2ban-regex`, and read the first lines of `spoofed-bots.log`.

To check a single address against the published ranges without running a script, use the [crawler IP verification tool](/tools/verify-bot-ip/). The operators' tokens and IP files are listed in the [AI crawler user agents list](/guides/ai-crawlers/ai-crawler-user-agents-list/).

## Official sources

- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Google, Verifying Googlebot and other Google crawlers: <https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests>
- Apple, About Applebot: <https://support.apple.com/en-us/119829>
- Common Crawl, CCBot and FAQ: <https://commoncrawl.org/ccbot> and <https://commoncrawl.org/faq>
- Microsoft, Which crawlers does Bing use: <https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0>
- DuckDuckGo, DuckAssistBot: <https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot>
- Mistral AI, Mistral crawlers: <https://docs.mistral.ai/robots>
- Amazon, Amazonbot: <https://developer.amazon.com/amazonbot>
- Meta, Meta Web Crawlers: <https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/>
- nginx, geo, map, realip and log modules: <https://nginx.org/en/docs/http/ngx_http_geo_module.html>, <https://nginx.org/en/docs/http/ngx_http_map_module.html>, <https://nginx.org/en/docs/http/ngx_http_realip_module.html>, <https://nginx.org/en/docs/http/ngx_http_log_module.html>
- fail2ban: <https://github.com/fail2ban/fail2ban>
