# AI Crawler User Agents List 2026 - GPTBot, ClaudeBot, PerplexityBot and robots.txt Tokens

> GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and more, with operator, robots.txt token, purpose, compliance and official IP ranges.

Source: https://iabig.com/guides/ai-crawlers/ai-crawler-user-agents-list/ — Guides CC BY 4.0, extraits de code MIT

AI companies no longer send a single crawler. They run several agents with distinct jobs: collecting pages for model training, indexing for a conversational search engine, or fetching a page on the spot because a user asked a question. Mixing them up leads you to block your visibility when you meant to block training, or the other way around. This list only covers agents described in their operator's official documentation, last checked on November 6, 2026. Links are collected at the end of the page.

## Three families to tell apart

- **Training**: the crawler collects pages that may be used to train models. Blocking it should, in principle, have no effect on whether you appear in search answers.
- **Search and indexing**: the crawler feeds the index that lets an assistant cite and link to your site. Blocking it reduces your visibility.
- **User-initiated actions**: the agent visits a page because a user asked it to (a link pasted into the chat, a live search, an agent task). Several operators state that robots.txt may not apply to these requests.

Two special cases: **Google-Extended** and **Applebot-Extended** are robots.txt tokens, not crawlers. No HTTP request ever carries these names. The crawling is done by Googlebot and Applebot, and the token only expresses a preference about how the data is used.

## Training crawlers

| robots.txt token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
| `GPTBot` | OpenAI | Content that may be used to train OpenAI's foundation models | Yes | `https://openai.com/gptbot.json` |
| `ClaudeBot` | Anthropic | Content that may contribute to model training | Yes, including `Crawl-delay` | `https://claude.com/crawling/bots.json` (one list shared by all Anthropic crawlers) |
| `CCBot` | Common Crawl | Open web archive that anyone can reuse | Yes (robots.txt read before each site, `Crawl-delay` honored) | `https://index.commoncrawl.org/ccbot.json` |
| `meta-externalagent` | Meta | Use cases "such as training foundation AI models or improving products by indexing content directly" | Meta says to add a `Disallow` to block it | No list published on Meta's page |
| `MistralAI-Training` | Mistral AI | Building training datasets | Yes, `Disallow` supported | No list published |
| `Amazonbot` | Amazon | Improving Amazon products; "may be used to train Amazon AI models" | Yes, but not `Crawl-delay` | `https://developer.amazon.com/amazonbot/ip-addresses/` (HTML page) |
| `Bytespider` | ByteDance | Not confirmed, no official documentation found | Not confirmed | Not confirmed |

Common Crawl does not present CCBot as a training crawler. It builds a public archive that anyone can download, including to train models. Blocking it is therefore the only way to keep your pages out of future archives.

As for Bytespider, the token shows up in every log file and blocklist, but we found no ByteDance page describing its purpose, its robots.txt behavior or its IP addresses. Claims about it come from third-party sources.

## Control tokens with no crawler behind them

| robots.txt token | Operator | What it controls | Effect on search |
|---|---|---|---|
| `Google-Extended` | Google | Whether content crawled by Google is used to train future Gemini models (Gemini Apps, Vertex AI API for Gemini) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI | According to Google, no effect on inclusion or ranking in Google Search |
| `Applebot-Extended` | Apple | Whether data crawled by Applebot is used to train Apple's foundation models | According to Apple, pages still appear in search results and the rule is not used for ranking |

A point that is often missed: disallowing `Google-Extended` also removes your content from grounding in the Gemini apps, not just from training. On Apple's side, opting out of generative answers in Siri and Search is done with the `nosnippet` meta tag, not with `Applebot-Extended`.

## Search and indexing crawlers

| robots.txt token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
| `OAI-SearchBot` | OpenAI | Surfacing websites in ChatGPT's search features | Yes; an excluded site can still appear as a navigational link | `https://openai.com/searchbot.json` |
| `Claude-SearchBot` | Anthropic | Indexing the web to improve the relevance of search answers | Yes | `https://claude.com/crawling/bots.json` |
| `PerplexityBot` | Perplexity | Surfacing and linking websites in Perplexity search results; not used for foundation models | Yes (documented robots.txt token; changes can take up to 24 hours) | `https://www.perplexity.ai/perplexitybot.json` |
| `Applebot` | Apple | Search in Spotlight, Siri and Safari; data may also be used for training unless you opt out with `Applebot-Extended` | Yes; follows Googlebot rules if Applebot is not named; ignores `Crawl-delay` | `https://search.developer.apple.com/applebot.json` |
| `Amzn-SearchBot` | Amazon | Amazon search experiences, including Alexa; no generative training | Yes; follows the rules given to other search bots if it is not named | `https://developer.amazon.com/amazonbot/searchbot-ip-addresses/` |
| `meta-webindexer` | Meta | Improving Meta AI search results | Meta encourages allowing it in order to be cited; compliance not specified | No list published |
| `MistralAI-Index` | Mistral AI | Indexing for Mistral's search; no training | Not specified by Mistral | `https://mistral.ai/mistralai-index-ips.json` |
| `DuckAssistBot` | DuckDuckGo | Real-time fetching for DuckDuckGo's AI-assisted answers; not used to train AI models | Yes; DuckDuckGo says a `Disallow` takes effect after 72 hours and does not affect organic rankings | `https://duckduckgo.com/duckassistbot.json` |

OpenAI notes that if both `GPTBot` and `OAI-SearchBot` are allowed, it may use the results of a single crawl for both purposes, so you will see fewer visits from one of the two in your logs.

## Agents acting on behalf of a user

| Token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
| `ChatGPT-User` | OpenAI | Visiting a page during a question in ChatGPT or a custom GPT, GPT Actions | "robots.txt rules may not apply"; not used to decide what appears in search | `https://openai.com/chatgpt-user.json` |
| `Claude-User` | Anthropic | Fetching a site when a user asks Claude a question | Anthropic states its bots honor robots.txt; blocking `Claude-User` may reduce your visibility in user-directed searches | `https://claude.com/crawling/bots.json` |
| `Perplexity-User` | Perplexity | Visiting a page to answer a question and cite it | "generally ignores robots.txt rules" | `https://www.perplexity.ai/perplexity-user.json` |
| `MistralAI-User` | Mistral AI | Visiting a page to answer a user in Mistral's assistant and cite the source; no automatic crawling | Documented token; compliance not specified | `https://mistral.ai/mistralai-user-ips.json` |
| `meta-externalfetcher` | Meta | On-demand link fetching, including for agent tasks | "may bypass robots.txt rules" | No list published |
| `Amzn-User` | Amazon | Live information to answer Alexa requests | "may not follow all robots.txt directives" | `https://developer.amazon.com/amazonbot/live-ip-addresses/` |
| `Google-Agent` (HTTP string) | Google | Google-hosted agents that browse and act on request | User-triggered fetchers generally ignore robots.txt; no documented robots.txt token | `https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json` |

Google also lists `Google-GeminiNotebook` (URLs that Gemini Notebook users add as sources; the former `Google-NotebookLM` token is supported until August 2026) in the same category of user-triggered fetchers. Anthropic, for its part, says that browsers it hosts on a user's behalf identify as `Claude-Agent`.

## Full published User-Agent strings

The following operators publish the exact string (Meta also documents a longer variant with a link to its documentation; Amazon and Google use `W.X.Y.Z` as a placeholder for the Chrome version). Version numbers can change, so match on the token, not on the whole string.

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
CCBot/2.0 (https://commoncrawl.org/faq/)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
meta-externalagent/1.1
meta-externalfetcher/1.1
meta-webindexer/1.1
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
```

OpenAI says it may add a `robots.txt` marker to the string when it fetches that file. Anthropic only publishes the names (`ClaudeBot`, `Claude-SearchBot`, `Claude-User`), not a full string. Older tokens such as `anthropic-ai` or `Claude-Web` still appear in many lists, but they are not in Anthropic's current documentation.

## Key takeaways

- A User-Agent string can be faked in one line. Only the published IP ranges (or reverse DNS for Google, Bing, Apple and Common Crawl) can authenticate a crawler. See [how to verify GPTBot, ClaudeBot and PerplexityBot](/guides/ai-crawlers/verify-gptbot-claudebot-perplexitybot/).
- Blocking a training crawler does not remove your site from search answers, with one exception, `Google-Extended`, which also covers grounding in Gemini.
- For user-initiated agents, robots.txt is not a guarantee. OpenAI, Perplexity, Meta, Amazon and Google say so explicitly; Mistral does not say either way.
- Anthropic points out that blocking its IP addresses is not a reliable way to opt out of training, because the crawler can then no longer read robots.txt.

Each crawler also has its own page in the [AI crawler directory](/bots/), for example [GPTBot](/bots/gptbot/) or [ClaudeBot](/bots/claudebot/). To see how widely these tokens are actually used, the weekly [AI crawler census](/ai-crawler-census/) tracks how the top 10,000 sites treat them in robots.txt and llms.txt. To check your own site, use the [IABIG checker](/).

## Official sources

- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Google, Google's common crawlers (Google-Extended): <https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers>
- Google, User-triggered fetchers: <https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers>
- Apple, About Applebot: <https://support.apple.com/en-us/119829>
- Common Crawl, CCBot and FAQ: <https://commoncrawl.org/ccbot> and <https://commoncrawl.org/faq>
- DuckDuckGo, DuckAssistBot: <https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot>
- Meta, Meta Web Crawlers: <https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/>
- Amazon, Amazonbot: <https://developer.amazon.com/amazonbot>
- Mistral AI, Mistral crawlers: <https://docs.mistral.ai/robots>
