How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNS
Confirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
Any script can send a User-Agent: GPTBot header. In a site's logs, some of the visits attributed to AI crawlers actually come from scrapers that borrow the name to get past filters, or simply to blend into the traffic. Common Crawl even warns about it on its own page: some bots pretend to be CCBot. The only way to settle the question is to compare the request's IP address with what the operator publishes. This guide gives the documented method for each crawler, ready-to-use commands, and then shows how to block impostors with nginx and fail2ban.
Which method for which crawler
Operators do not all use the same method. Most publish a JSON file of IP ranges. Google, Bing, Apple and Common Crawl also document verification by reverse DNS followed by a forward lookup.
| Crawler | Documented method | Source of truth |
|---|---|---|
GPTBot |
IP ranges | https://openai.com/gptbot.json |
OAI-SearchBot |
IP ranges | https://openai.com/searchbot.json |
ChatGPT-User |
IP ranges | https://openai.com/chatgpt-user.json |
ClaudeBot, Claude-SearchBot, Claude-User |
IP ranges (one list for all Anthropic crawlers) | https://claude.com/crawling/bots.json |
PerplexityBot |
IP ranges | https://www.perplexity.ai/perplexitybot.json |
Perplexity-User |
IP ranges | https://www.perplexity.ai/perplexity-user.json |
Googlebot and Google fetchers |
Reverse + forward DNS, or IP ranges | googlebot.com, google.com, googleusercontent.com |
Applebot |
Reverse DNS or IP ranges | *.applebot.apple.com, https://search.developer.apple.com/applebot.json |
CCBot |
IP ranges, reverse + forward DNS over IPv4 | *.crawl.commoncrawl.org, https://index.commoncrawl.org/ccbot.json |
Bingbot (also feeds Microsoft Copilot) |
Reverse + forward DNS, or IP ranges | *.search.msn.com, https://www.bing.com/toolbox/bingbot.json |
DuckAssistBot |
IP ranges | https://duckduckgo.com/duckassistbot.json |
MistralAI-User |
IP ranges | https://mistral.ai/mistralai-user-ips.json |
OpenAI, Anthropic, Perplexity, DuckDuckGo and Mistral do not document a reverse DNS hostname. For these crawlers, do not base a decision on DNS; use the JSON files. They share the same layout (the one Google and Bing also use): a top-level creationTime timestamp and a prefixes array in which each item holds an ipv4Prefix or ipv6Prefix in CIDR notation. Some files add extra fields, such as Common Crawl's synctoken and notes, which you can ignore.
{
"creationTime": "...",
"prefixes": [
{ "ipv4Prefix": "192.0.2.0/24" },
{ "ipv6Prefix": "2001:db8::/32" }
]
}
The ranges change, and some files are regenerated very often (the creationTime of OpenAI's chatgpt-user.json is frequently only hours old). Download them regularly instead of copying them once and for all.
Extracting the ranges with curl and jq
# CIDR prefixes published for GPTBot
curl -fsSL https://openai.com/gptbot.json | jq -r '.prefixes[] | .ipv4Prefix // .ipv6Prefix'
# File generation date
curl -fsSL https://claude.com/crawling/bots.json | jq -r '.creationTime'
# Number of prefixes per file
for u in https://openai.com/gptbot.json \
https://openai.com/searchbot.json \
https://claude.com/crawling/bots.json \
https://www.perplexity.ai/perplexitybot.json; do
printf '%5d %s\n' "$(curl -fsSL "$u" | jq '.prefixes | length')" "$u"
done
jq cannot test whether an address belongs to a CIDR network. For that step, Python and its standard ipaddress module are more reliable than a regular expression.
Testing an IP address with Python
The following script downloads the official lists and tells you which crawler an address belongs to. It only uses the standard library.
#!/usr/bin/env python3
"""Tells whether an IP address belongs to the published ranges of an AI crawler."""
import ipaddress
import json
import sys
import urllib.request
SOURCES = {
"GPTBot": "https://openai.com/gptbot.json",
"OAI-SearchBot": "https://openai.com/searchbot.json",
"ChatGPT-User": "https://openai.com/chatgpt-user.json",
"Anthropic (ClaudeBot, Claude-SearchBot, Claude-User)": "https://claude.com/crawling/bots.json",
"PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
"Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}
def prefixes(url):
req = urllib.request.Request(url, headers={"User-Agent": "verify-ip/1.0"})
with urllib.request.urlopen(req, timeout=15) as resp:
data = json.load(resp)
for p in data.get("prefixes", []):
cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
if cidr:
yield ipaddress.ip_network(cidr, strict=False)
def main():
if len(sys.argv) != 2:
sys.exit("usage: verify_bot.py <IP address>")
ip = ipaddress.ip_address(sys.argv[1])
found = False
for name, url in SOURCES.items():
try:
networks = list(prefixes(url))
except Exception as exc: # network error, invalid JSON...
print(f"[error] {name}: {exc}", file=sys.stderr)
continue
for net in networks:
if ip.version == net.version and ip in net:
print(f"{ip} belongs to {name} ({net})")
found = True
if not found:
print(f"{ip} is not in any of the published ranges checked")
sys.exit(0 if found else 1)
if __name__ == "__main__":
main()
python3 verify_bot.py 20.171.206.5
# 20.171.206.5 belongs to GPTBot (20.171.206.0/24) (result as of November 6, 2026)
# Check every IP that claims to be GPTBot in a log file
grep -i 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort -u |
while read -r ip; do python3 verify_bot.py "$ip" >/dev/null || echo "SUSPECT $ip"; done
The exit code (0 if the address is found, 1 otherwise) makes it easy to use in a script. For large volumes, load the lists once rather than rerunning the script for each address.
Mind the granularity: Anthropic's list covers all its crawlers without telling them apart. An IP on that list proves the request comes from Anthropic, not that it is ClaudeBot rather than Claude-User.
Reverse then forward DNS, where that is the documented method
For Google, Bing, Apple and Common Crawl, verification takes two steps: a reverse lookup of the IP to a hostname, a check of the domain, then a forward lookup of that name, which must return the same IP. The second step is essential, because whoever owns an IP address can set its PTR record to anything.
# Example published by Google
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
# Apple: the name must end in .applebot.apple.com
host 17.58.101.179
# Common Crawl: the name must end in .crawl.commoncrawl.org (IPv4 only)
dig +short -x 18.97.14.84
dig +short 18-97-14-84.crawl.commoncrawl.org A
The same logic in Python, for automation:
#!/usr/bin/env python3
"""Reverse then forward DNS verification (FCrDNS)."""
import ipaddress
import socket
import sys
DOMAINS = {
"googlebot": (".googlebot.com", ".google.com", ".googleusercontent.com"),
"applebot": (".applebot.apple.com",),
"ccbot": (".crawl.commoncrawl.org",),
"bingbot": (".search.msn.com",),
}
def fcrdns(ip, suffixes):
try:
name = socket.gethostbyaddr(ip)[0].rstrip(".")
except OSError:
return None, False
if not name.endswith(suffixes):
return name, False
try:
addresses = {ipaddress.ip_address(ai[4][0].split("%")[0])
for ai in socket.getaddrinfo(name, None)}
except OSError:
return name, False
return name, ipaddress.ip_address(ip) in addresses
if __name__ == "__main__":
if len(sys.argv) != 3 or sys.argv[1] not in DOMAINS:
sys.exit("usage: fcrdns.py googlebot|applebot|ccbot|bingbot <IP address>")
name, ok = fcrdns(sys.argv[2], DOMAINS[sys.argv[1]])
print(f"{sys.argv[2]} -> {name}: {'authentic' if ok else 'NOT verified'}")
sys.exit(0 if ok else 1)
The check is on the suffix preceded by a dot, so googlebot.com.example.net or fake-googlebot.com are rejected. Google notes that user-triggered fetchers resolve to google.com or gae.googleusercontent.com depending on the case. A googleusercontent.com name can therefore belong to a service hosted on Google Cloud rather than to Google itself.
Blocking impostors with nginx
The most robust approach is to let nginx compare, for every request, the crawler being claimed with the operator that owns the IP. First, generate a range file for the geo module:
#!/usr/bin/env bash
# /usr/local/sbin/update-ai-bot-ips.sh - run daily (cron or systemd timer)
set -euo pipefail
output=/etc/nginx/ai-bot-ips.conf
tmp=$(mktemp)
add() { # $1 = operator, $2 = URL of the official JSON
curl -fsSL "$2" | jq -r --arg v "$1" '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " " + $v + ";"' >> "$tmp"
}
add openai https://openai.com/gptbot.json
add openai https://openai.com/searchbot.json
add openai https://openai.com/chatgpt-user.json
add anthropic https://claude.com/crawling/bots.json
add perplexity https://www.perplexity.ai/perplexitybot.json
add perplexity https://www.perplexity.ai/perplexity-user.json
[ -s "$tmp" ] || { echo "empty list, aborting" >&2; exit 1; }
sort -u "$tmp" > "$output.new" && rm -f "$tmp"
mv "$output.new" "$output"
nginx -t && systemctl reload nginx
With set -e, a failed download stops the script and the previous list stays in place. The nginx configuration, in the http block:
# Operator that owns the IP according to the official lists
geo $ip_ai_operator {
default none;
include /etc/nginx/ai-bot-ips.conf;
}
# Operator claimed by the User-Agent
map $http_user_agent $ua_ai_operator {
default "";
"~*GPTBot|OAI-SearchBot|ChatGPT-User" openai;
"~*ClaudeBot|Claude-SearchBot|Claude-User" anthropic;
"~*PerplexityBot|Perplexity-User" perplexity;
}
# 1 = crawler claimed but IP outside its operator's ranges
map "$ua_ai_operator:$ip_ai_operator" $spoofed_bot {
default 1;
"~^:" 0; # no AI crawler claimed
"openai:openai" 0;
"anthropic:anthropic" 0;
"perplexity:perplexity" 0;
}
log_format spoofed_bots '$remote_addr [$time_local] "$request" "$http_user_agent"';
Then in the site's server block:
# An access_log directive in server replaces those inherited from http,
# so the main log is declared again.
access_log /var/log/nginx/access.log combined;
access_log /var/log/nginx/spoofed-bots.log spoofed_bots if=$spoofed_bot;
if ($spoofed_bot) {
return 403;
}
Genuine crawlers get through, impostors receive a 403 and are recorded in a dedicated log. To block a specific address right away, the standard directive is enough: deny 203.0.113.7; in the server block, or an included file that groups these lines.
If your site sits behind a CDN or a proxy, $remote_addr holds the proxy's address. Configure the realip module first (set_real_ip_from and real_ip_header) to recover the client IP, otherwise genuine crawlers will be classified as impostors.
Banning repeat offenders with fail2ban
The spoofed-bots.log file only contains requests already identified as fraudulent, so fail2ban can use it to ban the IP at the firewall level, which also protects the machine's other services.
# /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
[Definition]
failregex = ^<HOST> \[
ignoreregex =
# /etc/fail2ban/jail.d/nginx-spoofed-bots.local
[nginx-spoofed-bots]
enabled = true
filter = nginx-spoofed-bots
logpath = /var/log/nginx/spoofed-bots.log
port = http,https
maxretry = 3
findtime = 10m
bantime = 1d
fail2ban-regex /var/log/nginx/spoofed-bots.log /etc/fail2ban/filter.d/nginx-spoofed-bots.conf
systemctl reload fail2ban
fail2ban-client status nginx-spoofed-bots
Behind a CDN, a ban at the machine's firewall does nothing, since connections arrive from the CDN. Apply the rule at the CDN level instead.
Choosing a proportionate response
An impostor is not necessarily malicious. It may be a testing tool, a monitoring service or a developer checking their own robots.txt. Scale the response to the volume and behavior you observe.
| Situation | Proportionate response |
|---|---|
| A few isolated requests | Log them, do not block |
| Sustained crawling with a fake User-Agent | 403 via nginx, as above |
| Aggressive crawling from several addresses | fail2ban ban, or a rule at the CDN level |
| Excessive load from a genuine crawler | robots.txt (and Crawl-delay for crawlers that honor it, such as ClaudeBot and CCBot), then contact the operator |
For a genuine crawler that is too aggressive, IP blocking is not the right path. Anthropic, Amazon (amazonbot@amazon.com) and Meta (webmasters@meta.com) publish a contact address in their documentation. Anthropic asks you to write from an address on the affected domain, and Amazon asks you to include the domain names concerned.
Finally, watch the impostor log after each list update. A sudden rise in "genuine" crawlers classified as impostors almost always points to a stale list or a CDN problem, not an attack.
What not to do
- Block an operator's IP ranges to opt out of training. Anthropic says this method is not reliable, because it also prevents the crawler from reading robots.txt. Use robots.txt to express an opt-out.
- Rely on reverse DNS for OpenAI, Anthropic, Perplexity, DuckDuckGo or Mistral. None of their documentation provides for it.
- Copy the ranges once and for all. They change, and a stale list will eventually block a genuine crawler.
- Deploy without testing. An overly broad regular expression in
mapcan classify legitimate traffic as an impostor. Check the configuration withnginx -t, the filter withfail2ban-regex, and read the first lines ofspoofed-bots.log.
To check a single address against the published ranges without running a script, use the crawler IP verification tool. The operators' tokens and IP files are listed in the AI crawler user agents list.
Official sources
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google, Verifying Googlebot and other Google crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
- Apple, About Applebot: https://support.apple.com/en-us/119829
- Common Crawl, CCBot and FAQ: https://commoncrawl.org/ccbot and https://commoncrawl.org/faq
- Microsoft, Which crawlers does Bing use: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- Mistral AI, Mistral crawlers: https://docs.mistral.ai/robots
- Amazon, Amazonbot: https://developer.amazon.com/amazonbot
- Meta, Meta Web Crawlers: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- nginx, geo, map, realip and log modules: https://nginx.org/en/docs/http/ngx_http_geo_module.html, https://nginx.org/en/docs/http/ngx_http_map_module.html, https://nginx.org/en/docs/http/ngx_http_realip_module.html, https://nginx.org/en/docs/http/ngx_http_log_module.html
- fail2ban: https://github.com/fail2ban/fail2ban
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.
- ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers comparedWhat ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.