Block AI Training with robots.txt and Stay Visible in ChatGPT Search, Perplexity and Claude
An annotated robots.txt that blocks GPTBot, ClaudeBot or CCBot while staying in ChatGPT Search, Perplexity and Claude, plus the RFC 9309 pitfalls to avoid.
Now that the major AI companies run separate crawlers for training and for search, you can refuse to let your pages be used to train models while still being cited in ChatGPT Search, Perplexity or Claude. That takes a carefully written robots.txt, because a single badly built group is enough to block everything, or nothing. This guide provides an annotated file, explains what each choice implies and lists the common mistakes in light of RFC 9309, the standard for the Robots Exclusion Protocol.
Which crawler does what
| Goal | Tokens involved | Source |
|---|---|---|
| Opt out of training | GPTBot, ClaudeBot, CCBot, meta-externalagent, MistralAI-Training, Bytespider, Google-Extended, Applebot-Extended |
Each operator's documentation (none found for Bytespider) |
| Appear in ChatGPT Search | OAI-SearchBot |
OpenAI |
| Appear in Claude's answers | Claude-SearchBot (indexing) and Claude-User (on-demand retrieval) |
Anthropic |
| Appear in Perplexity | PerplexityBot |
Perplexity |
OpenAI states that each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. Perplexity describes the same principle for its agents.
The recommended robots.txt
# robots.txt - https://www.example.com/robots.txt
# Goal: opt out of training, stay visible in AI search.
# 1. Model training: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /
# 2. Search and on-demand retrieval: allowed
# These crawlers apply ONLY this group, so the exclusions from the * group are repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /cart/
Allow: /
# 3. All other crawlers (Googlebot, Bingbot, Applebot...)
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://www.example.com/sitemap.xml
A few notes on this file:
- Several consecutive
User-agentlines form a single group, which RFC 9309 explicitly allows. The file stays readable and each rule is written only once per group. Allow: /is not required (no matching rule means allowed), but it makes the intent explicit for anyone reviewing the file.Applebotis not named, so it follows the*group and keeps feeding Spotlight, Siri and Safari. The training opt-out is expressed byApplebot-Extended.Googlebotstays in the*group. According to Google,Google-Extendedhas no effect on inclusion or ranking in Google Search.Bytespideris blocked as a precaution. To our knowledge, ByteDance publishes no documentation about this crawler, and nothing guarantees that it reads this file.Perplexity-UserandChatGPT-Userare listed in the allowed group for consistency, but their operators state that robots.txt may not apply to these requests.Amazonbotis deliberately left out. Amazon says it improves Amazon products and services and "may" also be used to train Amazon AI models, with no separate token for the training part. Add it to the first group only if you accept that trade-off;Amzn-SearchBot, which feeds Amazon search experiences such as Alexa, is a separate token.
What each choice means
| Rule | Expected effect according to the operator | What you lose |
|---|---|---|
GPTBot disallowed |
Signals that the content should not be used to train OpenAI's models | Nothing in ChatGPT Search if OAI-SearchBot stays allowed |
OAI-SearchBot disallowed |
The site no longer appears in ChatGPT search answers | Citations; a navigational link may remain |
ClaudeBot disallowed |
Future content is excluded from Anthropic's training datasets | Nothing on the search side |
Claude-SearchBot disallowed |
Content is no longer indexed for search | Visibility and accuracy in Claude's results |
Claude-User disallowed |
Claude no longer fetches your pages on demand | Visibility in user-directed searches |
PerplexityBot disallowed |
The site is no longer surfaced in Perplexity results | Citations in Perplexity |
Google-Extended disallowed |
Excluded from Gemini training and from grounding in Gemini Apps and Vertex AI | Grounding in Gemini answers, not Google Search |
Applebot-Extended disallowed |
Excluded from training Apple's models | Nothing in Spotlight, Siri or Safari |
CCBot disallowed |
Your pages no longer enter future Common Crawl archives | Nothing on the AI engine side |
Three caveats deserve attention. First, an opt-out applies going forward. Anthropic, for example, says it signals the exclusion of future content, and none of the pages cited promises to remove what has already been collected. Second, changes are not instant. OpenAI mentions about 24 hours for search, Perplexity and Meta up to 24 hours, DuckDuckGo 72 hours, Amazon says Amazonbot may use a cached robots.txt up to 30 days old, and RFC 9309 says a crawler should not keep a cached copy for more than 24 hours unless the file is unreachable. Third, Anthropic asks you to repeat the process for every subdomain, since blog.example.com has its own robots.txt.
If you also want to stay visible in Gemini, do not disallow Google-Extended. Google states that this token covers both training and grounding in Gemini Apps. The current documentation offers no token that separates the two.
Common mistakes
Assuming a crawler combines its own group with the * group
This is the most common mistake. Under RFC 9309, a crawler looks for the group that carries its name. If it finds one, it applies that group alone. The * group is only used when no group matches.
User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
Allow: /
# OAI-SearchBot can read /admin/ because the * group no longer applies to it.
Rules are merged only when several groups name the same crawler. In that case the RFC requires combining them into one.
Relying on line order
Under the RFC, the order of groups does not matter (the crawler looks for its name), and neither does the order of rules. The most specific rule wins, meaning the one whose path has the most bytes. If an Allow and a Disallow tie, Allow should be used.
User-agent: PerplexityBot
Disallow: /docs/
Allow: /docs/public/
# /docs/public/guide.html: allowed (longer Allow)
# /docs/internal.html : disallowed
Even so, do not rely on any particular order to resolve an ambiguity. Write rules that do not contradict each other.
Confusing an empty Disallow: with Disallow: /
User-agent: GPTBot
Disallow:
# Empty value: no rule matches, EVERYTHING is allowed.
User-agent: ClaudeBot
Disallow: /
# The whole site is disallowed.
An empty Disallow: is the old way of writing "allow everything". Copied by mistake into a blocking group, it cancels the block.
Getting case sensitivity wrong
Matching on the crawler name is case-insensitive, so gptbot and GPTBot refer to the same group. Path matching, on the other hand, should be case-sensitive according to the RFC, so Disallow: /Admin/ does not protect /admin/. Check your site's actual URLs before writing a rule.
Misspelling a token
The name must match the published token, apart from case. Claude-Bot or GPT-Bot do not refer to any known crawler. The tokens anthropic-ai and Claude-Web, still copied into many templates, are not in Anthropic's current documentation. Use ClaudeBot, Claude-SearchBot and Claude-User instead.
Serving robots.txt with a 5xx error
The RFC distinguishes two cases. A 4xx code (file missing) lets the crawler crawl everything. A 5xx server error or a network failure must be treated as a complete disallow. A robots.txt that returns 500 behind a misconfigured firewall or CDN can therefore make you invisible to every crawler that follows the standard.
Forgetting what robots.txt does not do
- It is not access control. The RFC points out that listing a path makes it public. Protect private areas with authentication.
- User-initiated agents may ignore it, as OpenAI states for
ChatGPT-Userand Perplexity forPerplexity-User. - A crawler that spoofs a name will not honor it. Only IP filtering, checked against the published ranges, will stop it.
Testing the file
After publishing, check three things: that the file returns a 200 (curl -I https://www.example.com/robots.txt), that it is served as plain text, and that no firewall rule blocks it for crawlers. Python ships a basic parser that is handy for a quick check, but it does not follow RFC 9309 exactly. It uses the first matching rule rather than the most specific one. On the /docs/ example above, it answers "disallowed" for /docs/public/guide.html, whereas the RFC allows it. Pass it the bare token (GPTBot), not the full User-Agent string, which it cannot match to the group.
from urllib.robotparser import RobotFileParser
rp = RobotFileParser("https://www.example.com/robots.txt")
rp.read()
for agent in ("GPTBot", "OAI-SearchBot", "ClaudeBot", "Claude-SearchBot", "PerplexityBot"):
print(agent, rp.can_fetch(agent, "https://www.example.com/guide/"))
For the differences between each operator's crawlers, see the AI crawler user agents list. To build a file like the one above, use the robots.txt generator; to test your rules against a given crawler, use the robots.txt tester. If you want to know how other sites handle the same choice, the weekly AI crawler census shows which AI tokens the top 10,000 sites disallow.
Official sources
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google, Google's common crawlers (Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Apple, About Applebot: https://support.apple.com/en-us/119829
- Common Crawl, CCBot: https://commoncrawl.org/ccbot
- Meta, Meta Web Crawlers: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- Amazon, Amazonbot: https://developer.amazon.com/amazonbot
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- Mistral AI, Mistral crawlers: https://docs.mistral.ai/robots
- Python, urllib.robotparser: https://docs.python.org/3/library/urllib.robotparser.html
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.