IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

Block AI Training with robots.txt and Stay Visible in ChatGPT Search, Perplexity and Claude

An annotated robots.txt that blocks GPTBot, ClaudeBot or CCBot while staying in ChatGPT Search, Perplexity and Claude, plus the RFC 9309 pitfalls to avoid.

Now that the major AI companies run separate crawlers for training and for search, you can refuse to let your pages be used to train models while still being cited in ChatGPT Search, Perplexity or Claude. That takes a carefully written robots.txt, because a single badly built group is enough to block everything, or nothing. This guide provides an annotated file, explains what each choice implies and lists the common mistakes in light of RFC 9309, the standard for the Robots Exclusion Protocol.

Which crawler does what

Goal Tokens involved Source
Opt out of training GPTBot, ClaudeBot, CCBot, meta-externalagent, MistralAI-Training, Bytespider, Google-Extended, Applebot-Extended Each operator's documentation (none found for Bytespider)
Appear in ChatGPT Search OAI-SearchBot OpenAI
Appear in Claude's answers Claude-SearchBot (indexing) and Claude-User (on-demand retrieval) Anthropic
Appear in Perplexity PerplexityBot Perplexity

OpenAI states that each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. Perplexity describes the same principle for its agents.

# robots.txt - https://www.example.com/robots.txt
# Goal: opt out of training, stay visible in AI search.

# 1. Model training: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and on-demand retrieval: allowed
#    These crawlers apply ONLY this group, so the exclusions from the * group are repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /cart/
Allow: /

# 3. All other crawlers (Googlebot, Bingbot, Applebot...)
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

A few notes on this file:

  • Several consecutive User-agent lines form a single group, which RFC 9309 explicitly allows. The file stays readable and each rule is written only once per group.
  • Allow: / is not required (no matching rule means allowed), but it makes the intent explicit for anyone reviewing the file.
  • Applebot is not named, so it follows the * group and keeps feeding Spotlight, Siri and Safari. The training opt-out is expressed by Applebot-Extended.
  • Googlebot stays in the * group. According to Google, Google-Extended has no effect on inclusion or ranking in Google Search.
  • Bytespider is blocked as a precaution. To our knowledge, ByteDance publishes no documentation about this crawler, and nothing guarantees that it reads this file.
  • Perplexity-User and ChatGPT-User are listed in the allowed group for consistency, but their operators state that robots.txt may not apply to these requests.
  • Amazonbot is deliberately left out. Amazon says it improves Amazon products and services and "may" also be used to train Amazon AI models, with no separate token for the training part. Add it to the first group only if you accept that trade-off; Amzn-SearchBot, which feeds Amazon search experiences such as Alexa, is a separate token.

What each choice means

Rule Expected effect according to the operator What you lose
GPTBot disallowed Signals that the content should not be used to train OpenAI's models Nothing in ChatGPT Search if OAI-SearchBot stays allowed
OAI-SearchBot disallowed The site no longer appears in ChatGPT search answers Citations; a navigational link may remain
ClaudeBot disallowed Future content is excluded from Anthropic's training datasets Nothing on the search side
Claude-SearchBot disallowed Content is no longer indexed for search Visibility and accuracy in Claude's results
Claude-User disallowed Claude no longer fetches your pages on demand Visibility in user-directed searches
PerplexityBot disallowed The site is no longer surfaced in Perplexity results Citations in Perplexity
Google-Extended disallowed Excluded from Gemini training and from grounding in Gemini Apps and Vertex AI Grounding in Gemini answers, not Google Search
Applebot-Extended disallowed Excluded from training Apple's models Nothing in Spotlight, Siri or Safari
CCBot disallowed Your pages no longer enter future Common Crawl archives Nothing on the AI engine side

Three caveats deserve attention. First, an opt-out applies going forward. Anthropic, for example, says it signals the exclusion of future content, and none of the pages cited promises to remove what has already been collected. Second, changes are not instant. OpenAI mentions about 24 hours for search, Perplexity and Meta up to 24 hours, DuckDuckGo 72 hours, Amazon says Amazonbot may use a cached robots.txt up to 30 days old, and RFC 9309 says a crawler should not keep a cached copy for more than 24 hours unless the file is unreachable. Third, Anthropic asks you to repeat the process for every subdomain, since blog.example.com has its own robots.txt.

If you also want to stay visible in Gemini, do not disallow Google-Extended. Google states that this token covers both training and grounding in Gemini Apps. The current documentation offers no token that separates the two.

Common mistakes

Assuming a crawler combines its own group with the * group

This is the most common mistake. Under RFC 9309, a crawler looks for the group that carries its name. If it finds one, it applies that group alone. The * group is only used when no group matches.

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /
# OAI-SearchBot can read /admin/ because the * group no longer applies to it.

Rules are merged only when several groups name the same crawler. In that case the RFC requires combining them into one.

Relying on line order

Under the RFC, the order of groups does not matter (the crawler looks for its name), and neither does the order of rules. The most specific rule wins, meaning the one whose path has the most bytes. If an Allow and a Disallow tie, Allow should be used.

User-agent: PerplexityBot
Disallow: /docs/
Allow: /docs/public/
# /docs/public/guide.html: allowed (longer Allow)
# /docs/internal.html    : disallowed

Even so, do not rely on any particular order to resolve an ambiguity. Write rules that do not contradict each other.

Confusing an empty Disallow: with Disallow: /

User-agent: GPTBot
Disallow:
# Empty value: no rule matches, EVERYTHING is allowed.

User-agent: ClaudeBot
Disallow: /
# The whole site is disallowed.

An empty Disallow: is the old way of writing "allow everything". Copied by mistake into a blocking group, it cancels the block.

Getting case sensitivity wrong

Matching on the crawler name is case-insensitive, so gptbot and GPTBot refer to the same group. Path matching, on the other hand, should be case-sensitive according to the RFC, so Disallow: /Admin/ does not protect /admin/. Check your site's actual URLs before writing a rule.

Misspelling a token

The name must match the published token, apart from case. Claude-Bot or GPT-Bot do not refer to any known crawler. The tokens anthropic-ai and Claude-Web, still copied into many templates, are not in Anthropic's current documentation. Use ClaudeBot, Claude-SearchBot and Claude-User instead.

Serving robots.txt with a 5xx error

The RFC distinguishes two cases. A 4xx code (file missing) lets the crawler crawl everything. A 5xx server error or a network failure must be treated as a complete disallow. A robots.txt that returns 500 behind a misconfigured firewall or CDN can therefore make you invisible to every crawler that follows the standard.

Forgetting what robots.txt does not do

  • It is not access control. The RFC points out that listing a path makes it public. Protect private areas with authentication.
  • User-initiated agents may ignore it, as OpenAI states for ChatGPT-User and Perplexity for Perplexity-User.
  • A crawler that spoofs a name will not honor it. Only IP filtering, checked against the published ranges, will stop it.

Testing the file

After publishing, check three things: that the file returns a 200 (curl -I https://www.example.com/robots.txt), that it is served as plain text, and that no firewall rule blocks it for crawlers. Python ships a basic parser that is handy for a quick check, but it does not follow RFC 9309 exactly. It uses the first matching rule rather than the most specific one. On the /docs/ example above, it answers "disallowed" for /docs/public/guide.html, whereas the RFC allows it. Pass it the bare token (GPTBot), not the full User-Agent string, which it cannot match to the group.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://www.example.com/robots.txt")
rp.read()
for agent in ("GPTBot", "OAI-SearchBot", "ClaudeBot", "Claude-SearchBot", "PerplexityBot"):
    print(agent, rp.can_fetch(agent, "https://www.example.com/guide/"))

For the differences between each operator's crawlers, see the AI crawler user agents list. To build a file like the one above, use the robots.txt generator; to test your rules against a given crawler, use the robots.txt tester. If you want to know how other sites handle the same choice, the weekly AI crawler census shows which AI tokens the top 10,000 sites disallow.

Official sources

Free tools

Related guides