IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers compared

What GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.

OpenAI sends three different agents to websites, and they do very different jobs. GPTBot collects content that may be used to train models, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT about it. OpenAI documents each one separately and says their robots.txt settings are independent, so you can refuse training without disappearing from ChatGPT search. This guide compares the three, explains what blocking each one does, gives ready-to-use robots.txt recipes and shows how to tell them apart in your server logs. Everything below is based on OpenAI's crawler documentation as of November 2026.

The three agents at a glance

GPTBot OAI-SearchBot ChatGPT-User
Purpose Content that may be used to train OpenAI's generative AI foundation models Surfacing websites in ChatGPT's search features Visiting a page during a user action in ChatGPT or a custom GPT
Crawls automatically Yes Yes No
Follows robots.txt Yes Yes OpenAI says robots.txt rules may not apply
User-agent token GPTBot OAI-SearchBot ChatGPT-User
Official IP list openai.com/gptbot.json openai.com/searchbot.json openai.com/chatgpt-user.json
Effect of blocking Opt-out of training Out of ChatGPT search answers Not guaranteed; no effect on search

Each agent has its own page in our directory: GPTBot, OAI-SearchBot and ChatGPT-User.

GPTBot, the training crawler

GPTBot crawls the web for content that may be used to train OpenAI's generative AI foundation models. OpenAI says that disallowing it in robots.txt signals that a site's content should not be used for that training.

Two details matter in practice. First, the setting is independent of OAI-SearchBot: blocking GPTBot does not remove you from ChatGPT search. Second, when both GPTBot and OAI-SearchBot are allowed, OpenAI says it may use the results of a single crawl for both purposes to avoid duplicate crawling. If you allow both, one of them may show up less often in your logs than you expect, and that is not a sign of a problem.

What OpenAI does not say is just as important. Its documentation does not explain what happens to content collected before you added a Disallow rule, and it does not give a delay for GPTBot to pick up a change (the roughly 24-hour figure is stated for search).

OAI-SearchBot, the ChatGPT search crawler

OAI-SearchBot is used to surface websites, with links, in ChatGPT's search features. According to OpenAI, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links. To appear in search results, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.

OpenAI also says that a robots.txt change can take about 24 hours to be reflected in search results. Of the three agents, this is the one whose blocking costs the most visibility: if being cited in ChatGPT matters to you, OAI-SearchBot must be allowed both in robots.txt and in your firewall or CDN.

The documentation does not say whether content fetched by OAI-SearchBot can also be used for training. It only notes that a single crawl may serve both purposes when GPTBot is allowed too.

ChatGPT-User, the on-demand agent

ChatGPT-User is not an automatic crawler. OpenAI uses it for certain user actions in ChatGPT and custom GPTs: when someone asks a question, ChatGPT may visit a web page with this agent. It is also used when users interact with external applications through GPT Actions.

OpenAI states three things about it:

  • Because these actions are initiated by a user, robots.txt rules may not apply.
  • It is not used to crawl the web in an automatic fashion.
  • It is not used to determine whether content may appear in search. Search visibility is managed with OAI-SearchBot.

So a Disallow rule for ChatGPT-User is a request with no guarantee behind it, and it never takes you out of ChatGPT search. If you really need to refuse these visits, you have to do it at the server or firewall level, which also means ChatGPT can no longer read your page when a user explicitly asks it to.

OpenAI's crawler page also lists a fourth agent, OAI-AdsBot, which checks landing pages submitted as ChatGPT ads. OpenAI says it only visits submitted ad pages and that its data is not used to train generative AI foundation models. It has no effect on organic visibility.

Decision table

Your goal GPTBot OAI-SearchBot ChatGPT-User
Maximum visibility in ChatGPT Allow Allow Allow
Refuse training, stay in ChatGPT search Disallow Allow Allow
Stay out of ChatGPT search as well Disallow Disallow Disallow in robots.txt, and block by IP if you need enforcement
Keep a private area out of all three Disallow the path Disallow the path Protect the path with authentication

The second row is what most publishers want. It is also the one OpenAI explicitly supports: its documentation describes the two settings as independent.

robots.txt recipes

Under RFC 9309, a crawler that finds a group naming it applies that group only and ignores the * group. If your * group excludes paths such as /admin/, repeat those exclusions in every named group.

# OpenAI training: refused
User-agent: GPTBot
Disallow: /

# ChatGPT search and user-initiated visits: allowed
# Named groups ignore the * group, so repeat its exclusions here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /admin/
Allow: /

User-agent: *
Disallow: /admin/

Refuse all three

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

Remember that the ChatGPT-User line is only a request. Your site will also stop appearing in ChatGPT search answers, apart from possible navigational links.

Allow everything except a private area

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /account/
Disallow: /checkout/

A path listed in robots.txt is public information, and ChatGPT-User may not honor it anyway. Anything truly private belongs behind a login.

Telling them apart in your logs

Each agent sends its own token in the user-agent header. OpenAI's published strings are:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Version numbers change, and the OAI-SearchBot string starts like a regular Chrome browser, so match on the token rather than the full string. OpenAI also says it may add a robots.txt marker to the user-agent string when fetching robots.txt, which lets you separate those requests from page fetches.

A quick count per agent in an nginx or Apache access log:

for t in GPTBot OAI-SearchBot ChatGPT-User; do
  printf '%8d  %s\n' "$(grep -ci "$t/" /var/log/nginx/access.log)" "$t"
done

Searching for the token followed by a slash (GPTBot/) counts each request once. A plain grep -o gptbot would count every GPTBot request twice, because its user-agent string also ends with the URL https://openai.com/gptbot.

The user-agent header is trivial to fake. OpenAI documents no reverse DNS hostname for these agents, so the only supported check is to compare the source IP with the list that matches the claimed agent. Each list uses the common prefixes JSON format with a creationTime field, and the ChatGPT-User list is long and regenerated often, so download the files on a schedule instead of copying them once.

import ipaddress, json, urllib.request

LISTS = {
    "gptbot": "https://openai.com/gptbot.json",
    "oai-searchbot": "https://openai.com/searchbot.json",
    "chatgpt-user": "https://openai.com/chatgpt-user.json",
}

def is_genuine(token, ip):
    """True if ip is in the official list for the agent it claims to be."""
    req = urllib.request.Request(LISTS[token.lower()], headers={"User-Agent": "ip-check/1.0"})
    with urllib.request.urlopen(req, timeout=10) as r:
        prefixes = json.load(r)["prefixes"]
    addr = ipaddress.ip_address(ip)
    return any(
        addr in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in prefixes
    )

print(is_genuine("GPTBot", "20.171.206.10"))

For a one-off check without code, paste the address into the crawler IP verification tool. Our guide on verifying GPTBot, ClaudeBot and PerplexityBot covers firewall rules, and measuring AI crawler traffic in server logs goes further on log analysis.

What OpenAI does not confirm

  • What happens to content GPTBot collected before you blocked it.
  • How long GPTBot takes to apply a robots.txt change (about 24 hours is stated for search only).
  • Whether content fetched by OAI-SearchBot can be used for training.
  • When exactly ChatGPT-User follows or ignores robots.txt, and whether pages it fetches are retained.

For the same comparison on the Anthropic side, see ClaudeBot vs Claude-SearchBot vs Claude-User. To opt out of training across every major company, see how to opt out of AI training.

Official sources

Free tools

Related guides