# Block AI Training with robots.txt and Stay Visible in ChatGPT Search, Perplexity and Claude

> An annotated robots.txt that blocks GPTBot, ClaudeBot or CCBot while staying in ChatGPT Search, Perplexity and Claude, plus the RFC 9309 pitfalls to avoid.

Source: https://iabig.com/guides/ai-crawlers/block-ai-training-robots-txt/ — Guides CC BY 4.0, extraits de code MIT

Now that the major AI companies run separate crawlers for training and for search, you can refuse to let your pages be used to train models while still being cited in ChatGPT Search, Perplexity or Claude. That takes a carefully written robots.txt, because a single badly built group is enough to block everything, or nothing. This guide provides an annotated file, explains what each choice implies and lists the common mistakes in light of RFC 9309, the standard for the Robots Exclusion Protocol.

## Which crawler does what

| Goal | Tokens involved | Source |
|---|---|---|
| Opt out of training | `GPTBot`, `ClaudeBot`, `CCBot`, `meta-externalagent`, `MistralAI-Training`, `Bytespider`, `Google-Extended`, `Applebot-Extended` | Each operator's documentation (none found for `Bytespider`) |
| Appear in ChatGPT Search | `OAI-SearchBot` | OpenAI |
| Appear in Claude's answers | `Claude-SearchBot` (indexing) and `Claude-User` (on-demand retrieval) | Anthropic |
| Appear in Perplexity | `PerplexityBot` | Perplexity |

OpenAI states that each setting is independent, so you can allow `OAI-SearchBot` and disallow `GPTBot`. Perplexity describes the same principle for its agents.

## The recommended robots.txt

```text
# robots.txt - https://www.example.com/robots.txt
# Goal: opt out of training, stay visible in AI search.

# 1. Model training: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and on-demand retrieval: allowed
#    These crawlers apply ONLY this group, so the exclusions from the * group are repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /cart/
Allow: /

# 3. All other crawlers (Googlebot, Bingbot, Applebot...)
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml
```

A few notes on this file:

- Several consecutive `User-agent` lines form a single group, which RFC 9309 explicitly allows. The file stays readable and each rule is written only once per group.
- `Allow: /` is not required (no matching rule means allowed), but it makes the intent explicit for anyone reviewing the file.
- `Applebot` is not named, so it follows the `*` group and keeps feeding Spotlight, Siri and Safari. The training opt-out is expressed by `Applebot-Extended`.
- `Googlebot` stays in the `*` group. According to Google, `Google-Extended` has no effect on inclusion or ranking in Google Search.
- `Bytespider` is blocked as a precaution. To our knowledge, ByteDance publishes no documentation about this crawler, and nothing guarantees that it reads this file.
- `Perplexity-User` and `ChatGPT-User` are listed in the allowed group for consistency, but their operators state that robots.txt may not apply to these requests.
- `Amazonbot` is deliberately left out. Amazon says it improves Amazon products and services and "may" also be used to train Amazon AI models, with no separate token for the training part. Add it to the first group only if you accept that trade-off; `Amzn-SearchBot`, which feeds Amazon search experiences such as Alexa, is a separate token.

## What each choice means

| Rule | Expected effect according to the operator | What you lose |
|---|---|---|
| `GPTBot` disallowed | Signals that the content should not be used to train OpenAI's models | Nothing in ChatGPT Search if `OAI-SearchBot` stays allowed |
| `OAI-SearchBot` disallowed | The site no longer appears in ChatGPT search answers | Citations; a navigational link may remain |
| `ClaudeBot` disallowed | Future content is excluded from Anthropic's training datasets | Nothing on the search side |
| `Claude-SearchBot` disallowed | Content is no longer indexed for search | Visibility and accuracy in Claude's results |
| `Claude-User` disallowed | Claude no longer fetches your pages on demand | Visibility in user-directed searches |
| `PerplexityBot` disallowed | The site is no longer surfaced in Perplexity results | Citations in Perplexity |
| `Google-Extended` disallowed | Excluded from Gemini training and from grounding in Gemini Apps and Vertex AI | Grounding in Gemini answers, not Google Search |
| `Applebot-Extended` disallowed | Excluded from training Apple's models | Nothing in Spotlight, Siri or Safari |
| `CCBot` disallowed | Your pages no longer enter future Common Crawl archives | Nothing on the AI engine side |

Three caveats deserve attention. First, an opt-out applies going forward. Anthropic, for example, says it signals the exclusion of future content, and none of the pages cited promises to remove what has already been collected. Second, changes are not instant. OpenAI mentions about 24 hours for search, Perplexity and Meta up to 24 hours, DuckDuckGo 72 hours, Amazon says Amazonbot may use a cached robots.txt up to 30 days old, and RFC 9309 says a crawler should not keep a cached copy for more than 24 hours unless the file is unreachable. Third, Anthropic asks you to repeat the process for every subdomain, since `blog.example.com` has its own robots.txt.

If you also want to stay visible in Gemini, do not disallow `Google-Extended`. Google states that this token covers both training and grounding in Gemini Apps. The current documentation offers no token that separates the two.

## Common mistakes

### Assuming a crawler combines its own group with the `*` group

This is the most common mistake. Under RFC 9309, a crawler looks for the group that carries its name. If it finds one, it applies that group **alone**. The `*` group is only used when no group matches.

```text
User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /
# OAI-SearchBot can read /admin/ because the * group no longer applies to it.
```

Rules are merged only when several groups name the same crawler. In that case the RFC requires combining them into one.

### Relying on line order

Under the RFC, the order of groups does not matter (the crawler looks for its name), and neither does the order of rules. The **most specific** rule wins, meaning the one whose path has the most bytes. If an `Allow` and a `Disallow` tie, `Allow` should be used.

```text
User-agent: PerplexityBot
Disallow: /docs/
Allow: /docs/public/
# /docs/public/guide.html: allowed (longer Allow)
# /docs/internal.html    : disallowed
```

Even so, do not rely on any particular order to resolve an ambiguity. Write rules that do not contradict each other.

### Confusing an empty `Disallow:` with `Disallow: /`

```text
User-agent: GPTBot
Disallow:
# Empty value: no rule matches, EVERYTHING is allowed.

User-agent: ClaudeBot
Disallow: /
# The whole site is disallowed.
```

An empty `Disallow:` is the old way of writing "allow everything". Copied by mistake into a blocking group, it cancels the block.

### Getting case sensitivity wrong

Matching on the crawler name is **case-insensitive**, so `gptbot` and `GPTBot` refer to the same group. Path matching, on the other hand, **should be** case-sensitive according to the RFC, so `Disallow: /Admin/` does not protect `/admin/`. Check your site's actual URLs before writing a rule.

### Misspelling a token

The name must match the published token, apart from case. `Claude-Bot` or `GPT-Bot` do not refer to any known crawler. The tokens `anthropic-ai` and `Claude-Web`, still copied into many templates, are not in Anthropic's current documentation. Use `ClaudeBot`, `Claude-SearchBot` and `Claude-User` instead.

### Serving robots.txt with a 5xx error

The RFC distinguishes two cases. A 4xx code (file missing) lets the crawler crawl everything. A 5xx server error or a network failure must be treated as a **complete disallow**. A robots.txt that returns 500 behind a misconfigured firewall or CDN can therefore make you invisible to every crawler that follows the standard.

### Forgetting what robots.txt does not do

- It is not access control. The RFC points out that listing a path makes it public. Protect private areas with authentication.
- User-initiated agents may ignore it, as OpenAI states for `ChatGPT-User` and Perplexity for `Perplexity-User`.
- A crawler that spoofs a name will not honor it. Only IP filtering, checked against the published ranges, will stop it.

## Testing the file

After publishing, check three things: that the file returns a 200 (`curl -I https://www.example.com/robots.txt`), that it is served as plain text, and that no firewall rule blocks it for crawlers. Python ships a basic parser that is handy for a quick check, but it does not follow RFC 9309 exactly. It uses the first matching rule rather than the most specific one. On the `/docs/` example above, it answers "disallowed" for `/docs/public/guide.html`, whereas the RFC allows it. Pass it the bare token (`GPTBot`), not the full User-Agent string, which it cannot match to the group.

```python
from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://www.example.com/robots.txt")
rp.read()
for agent in ("GPTBot", "OAI-SearchBot", "ClaudeBot", "Claude-SearchBot", "PerplexityBot"):
    print(agent, rp.can_fetch(agent, "https://www.example.com/guide/"))
```

For the differences between each operator's crawlers, see the [AI crawler user agents list](/guides/ai-crawlers/ai-crawler-user-agents-list/). To build a file like the one above, use the [robots.txt generator](/tools/robots-txt-generator/); to test your rules against a given crawler, use the [robots.txt tester](/tools/robots-txt-tester/). If you want to know how other sites handle the same choice, the weekly [AI crawler census](/ai-crawler-census/) shows which AI tokens the top 10,000 sites disallow.

## Official sources

- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Google, Google's common crawlers (Google-Extended): <https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers>
- Apple, About Applebot: <https://support.apple.com/en-us/119829>
- Common Crawl, CCBot: <https://commoncrawl.org/ccbot>
- Meta, Meta Web Crawlers: <https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/>
- Amazon, Amazonbot: <https://developer.amazon.com/amazonbot>
- DuckDuckGo, DuckAssistBot: <https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot>
- Mistral AI, Mistral crawlers: <https://docs.mistral.ai/robots>
- Python, urllib.robotparser: <https://docs.python.org/3/library/urllib.robotparser.html>
