GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers compared
What GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.
OpenAI sends three different agents to websites, and they do very different jobs. GPTBot collects content that may be used to train models, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT about it. OpenAI documents each one separately and says their robots.txt settings are independent, so you can refuse training without disappearing from ChatGPT search. This guide compares the three, explains what blocking each one does, gives ready-to-use robots.txt recipes and shows how to tell them apart in your server logs. Everything below is based on OpenAI's crawler documentation as of November 2026.
The three agents at a glance
| GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Purpose | Content that may be used to train OpenAI's generative AI foundation models | Surfacing websites in ChatGPT's search features | Visiting a page during a user action in ChatGPT or a custom GPT |
| Crawls automatically | Yes | Yes | No |
| Follows robots.txt | Yes | Yes | OpenAI says robots.txt rules may not apply |
| User-agent token | GPTBot |
OAI-SearchBot |
ChatGPT-User |
| Official IP list | openai.com/gptbot.json |
openai.com/searchbot.json |
openai.com/chatgpt-user.json |
| Effect of blocking | Opt-out of training | Out of ChatGPT search answers | Not guaranteed; no effect on search |
Each agent has its own page in our directory: GPTBot, OAI-SearchBot and ChatGPT-User.
GPTBot, the training crawler
GPTBot crawls the web for content that may be used to train OpenAI's generative AI foundation models. OpenAI says that disallowing it in robots.txt signals that a site's content should not be used for that training.
Two details matter in practice. First, the setting is independent of OAI-SearchBot: blocking GPTBot does not remove you from ChatGPT search. Second, when both GPTBot and OAI-SearchBot are allowed, OpenAI says it may use the results of a single crawl for both purposes to avoid duplicate crawling. If you allow both, one of them may show up less often in your logs than you expect, and that is not a sign of a problem.
What OpenAI does not say is just as important. Its documentation does not explain what happens to content collected before you added a Disallow rule, and it does not give a delay for GPTBot to pick up a change (the roughly 24-hour figure is stated for search).
OAI-SearchBot, the ChatGPT search crawler
OAI-SearchBot is used to surface websites, with links, in ChatGPT's search features. According to OpenAI, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links. To appear in search results, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.
OpenAI also says that a robots.txt change can take about 24 hours to be reflected in search results. Of the three agents, this is the one whose blocking costs the most visibility: if being cited in ChatGPT matters to you, OAI-SearchBot must be allowed both in robots.txt and in your firewall or CDN.
The documentation does not say whether content fetched by OAI-SearchBot can also be used for training. It only notes that a single crawl may serve both purposes when GPTBot is allowed too.
ChatGPT-User, the on-demand agent
ChatGPT-User is not an automatic crawler. OpenAI uses it for certain user actions in ChatGPT and custom GPTs: when someone asks a question, ChatGPT may visit a web page with this agent. It is also used when users interact with external applications through GPT Actions.
OpenAI states three things about it:
- Because these actions are initiated by a user, robots.txt rules may not apply.
- It is not used to crawl the web in an automatic fashion.
- It is not used to determine whether content may appear in search. Search visibility is managed with OAI-SearchBot.
So a Disallow rule for ChatGPT-User is a request with no guarantee behind it, and it never takes you out of ChatGPT search. If you really need to refuse these visits, you have to do it at the server or firewall level, which also means ChatGPT can no longer read your page when a user explicitly asks it to.
OpenAI's crawler page also lists a fourth agent, OAI-AdsBot, which checks landing pages submitted as ChatGPT ads. OpenAI says it only visits submitted ad pages and that its data is not used to train generative AI foundation models. It has no effect on organic visibility.
Decision table
| Your goal | GPTBot | OAI-SearchBot | ChatGPT-User |
|---|---|---|---|
| Maximum visibility in ChatGPT | Allow | Allow | Allow |
| Refuse training, stay in ChatGPT search | Disallow | Allow | Allow |
| Stay out of ChatGPT search as well | Disallow | Disallow | Disallow in robots.txt, and block by IP if you need enforcement |
| Keep a private area out of all three | Disallow the path | Disallow the path | Protect the path with authentication |
The second row is what most publishers want. It is also the one OpenAI explicitly supports: its documentation describes the two settings as independent.
robots.txt recipes
Under RFC 9309, a crawler that finds a group naming it applies that group only and ignores the * group. If your * group excludes paths such as /admin/, repeat those exclusions in every named group.
Refuse training, stay in ChatGPT search
# OpenAI training: refused
User-agent: GPTBot
Disallow: /
# ChatGPT search and user-initiated visits: allowed
# Named groups ignore the * group, so repeat its exclusions here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /admin/
Allow: /
User-agent: *
Disallow: /admin/
Refuse all three
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /
Remember that the ChatGPT-User line is only a request. Your site will also stop appearing in ChatGPT search answers, apart from possible navigational links.
Allow everything except a private area
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /account/
Disallow: /checkout/
A path listed in robots.txt is public information, and ChatGPT-User may not honor it anyway. Anything truly private belongs behind a login.
Telling them apart in your logs
Each agent sends its own token in the user-agent header. OpenAI's published strings are:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Version numbers change, and the OAI-SearchBot string starts like a regular Chrome browser, so match on the token rather than the full string. OpenAI also says it may add a robots.txt marker to the user-agent string when fetching robots.txt, which lets you separate those requests from page fetches.
A quick count per agent in an nginx or Apache access log:
for t in GPTBot OAI-SearchBot ChatGPT-User; do
printf '%8d %s\n' "$(grep -ci "$t/" /var/log/nginx/access.log)" "$t"
done
Searching for the token followed by a slash (GPTBot/) counts each request once. A plain grep -o gptbot would count every GPTBot request twice, because its user-agent string also ends with the URL https://openai.com/gptbot.
The user-agent header is trivial to fake. OpenAI documents no reverse DNS hostname for these agents, so the only supported check is to compare the source IP with the list that matches the claimed agent. Each list uses the common prefixes JSON format with a creationTime field, and the ChatGPT-User list is long and regenerated often, so download the files on a schedule instead of copying them once.
import ipaddress, json, urllib.request
LISTS = {
"gptbot": "https://openai.com/gptbot.json",
"oai-searchbot": "https://openai.com/searchbot.json",
"chatgpt-user": "https://openai.com/chatgpt-user.json",
}
def is_genuine(token, ip):
"""True if ip is in the official list for the agent it claims to be."""
req = urllib.request.Request(LISTS[token.lower()], headers={"User-Agent": "ip-check/1.0"})
with urllib.request.urlopen(req, timeout=10) as r:
prefixes = json.load(r)["prefixes"]
addr = ipaddress.ip_address(ip)
return any(
addr in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
for p in prefixes
)
print(is_genuine("GPTBot", "20.171.206.10"))
For a one-off check without code, paste the address into the crawler IP verification tool. Our guide on verifying GPTBot, ClaudeBot and PerplexityBot covers firewall rules, and measuring AI crawler traffic in server logs goes further on log analysis.
What OpenAI does not confirm
- What happens to content GPTBot collected before you blocked it.
- How long GPTBot takes to apply a robots.txt change (about 24 hours is stated for search only).
- Whether content fetched by OAI-SearchBot can be used for training.
- When exactly ChatGPT-User follows or ignores robots.txt, and whether pages it fetches are retained.
For the same comparison on the Anthropic side, see ClaudeBot vs Claude-SearchBot vs Claude-User. To opt out of training across every major company, see how to opt out of AI training.
Official sources
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- OpenAI, GPTBot IP ranges: https://openai.com/gptbot.json
- OpenAI, OAI-SearchBot IP ranges: https://openai.com/searchbot.json
- OpenAI, ChatGPT-User IP ranges: https://openai.com/chatgpt-user.json
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers comparedWhat ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.