AI Crawler User Agents List 2026 - GPTBot, ClaudeBot, PerplexityBot and robots.txt Tokens
GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and more, with operator, robots.txt token, purpose, compliance and official IP ranges.
AI companies no longer send a single crawler. They run several agents with distinct jobs: collecting pages for model training, indexing for a conversational search engine, or fetching a page on the spot because a user asked a question. Mixing them up leads you to block your visibility when you meant to block training, or the other way around. This list only covers agents described in their operator's official documentation, last checked on November 6, 2026. Links are collected at the end of the page.
Three families to tell apart
- Training: the crawler collects pages that may be used to train models. Blocking it should, in principle, have no effect on whether you appear in search answers.
- Search and indexing: the crawler feeds the index that lets an assistant cite and link to your site. Blocking it reduces your visibility.
- User-initiated actions: the agent visits a page because a user asked it to (a link pasted into the chat, a live search, an agent task). Several operators state that robots.txt may not apply to these requests.
Two special cases: Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. No HTTP request ever carries these names. The crawling is done by Googlebot and Applebot, and the token only expresses a preference about how the data is used.
Training crawlers
| robots.txt token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
GPTBot |
OpenAI | Content that may be used to train OpenAI's foundation models | Yes | https://openai.com/gptbot.json |
ClaudeBot |
Anthropic | Content that may contribute to model training | Yes, including Crawl-delay |
https://claude.com/crawling/bots.json (one list shared by all Anthropic crawlers) |
CCBot |
Common Crawl | Open web archive that anyone can reuse | Yes (robots.txt read before each site, Crawl-delay honored) |
https://index.commoncrawl.org/ccbot.json |
meta-externalagent |
Meta | Use cases "such as training foundation AI models or improving products by indexing content directly" | Meta says to add a Disallow to block it |
No list published on Meta's page |
MistralAI-Training |
Mistral AI | Building training datasets | Yes, Disallow supported |
No list published |
Amazonbot |
Amazon | Improving Amazon products; "may be used to train Amazon AI models" | Yes, but not Crawl-delay |
https://developer.amazon.com/amazonbot/ip-addresses/ (HTML page) |
Bytespider |
ByteDance | Not confirmed, no official documentation found | Not confirmed | Not confirmed |
Common Crawl does not present CCBot as a training crawler. It builds a public archive that anyone can download, including to train models. Blocking it is therefore the only way to keep your pages out of future archives.
As for Bytespider, the token shows up in every log file and blocklist, but we found no ByteDance page describing its purpose, its robots.txt behavior or its IP addresses. Claims about it come from third-party sources.
Control tokens with no crawler behind them
| robots.txt token | Operator | What it controls | Effect on search |
|---|---|---|---|
Google-Extended |
Whether content crawled by Google is used to train future Gemini models (Gemini Apps, Vertex AI API for Gemini) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI | According to Google, no effect on inclusion or ranking in Google Search | |
Applebot-Extended |
Apple | Whether data crawled by Applebot is used to train Apple's foundation models | According to Apple, pages still appear in search results and the rule is not used for ranking |
A point that is often missed: disallowing Google-Extended also removes your content from grounding in the Gemini apps, not just from training. On Apple's side, opting out of generative answers in Siri and Search is done with the nosnippet meta tag, not with Applebot-Extended.
Search and indexing crawlers
| robots.txt token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
OAI-SearchBot |
OpenAI | Surfacing websites in ChatGPT's search features | Yes; an excluded site can still appear as a navigational link | https://openai.com/searchbot.json |
Claude-SearchBot |
Anthropic | Indexing the web to improve the relevance of search answers | Yes | https://claude.com/crawling/bots.json |
PerplexityBot |
Perplexity | Surfacing and linking websites in Perplexity search results; not used for foundation models | Yes (documented robots.txt token; changes can take up to 24 hours) | https://www.perplexity.ai/perplexitybot.json |
Applebot |
Apple | Search in Spotlight, Siri and Safari; data may also be used for training unless you opt out with Applebot-Extended |
Yes; follows Googlebot rules if Applebot is not named; ignores Crawl-delay |
https://search.developer.apple.com/applebot.json |
Amzn-SearchBot |
Amazon | Amazon search experiences, including Alexa; no generative training | Yes; follows the rules given to other search bots if it is not named | https://developer.amazon.com/amazonbot/searchbot-ip-addresses/ |
meta-webindexer |
Meta | Improving Meta AI search results | Meta encourages allowing it in order to be cited; compliance not specified | No list published |
MistralAI-Index |
Mistral AI | Indexing for Mistral's search; no training | Not specified by Mistral | https://mistral.ai/mistralai-index-ips.json |
DuckAssistBot |
DuckDuckGo | Real-time fetching for DuckDuckGo's AI-assisted answers; not used to train AI models | Yes; DuckDuckGo says a Disallow takes effect after 72 hours and does not affect organic rankings |
https://duckduckgo.com/duckassistbot.json |
OpenAI notes that if both GPTBot and OAI-SearchBot are allowed, it may use the results of a single crawl for both purposes, so you will see fewer visits from one of the two in your logs.
Agents acting on behalf of a user
| Token | Operator | Stated purpose | Honors robots.txt (per operator) | Published IP ranges |
|---|---|---|---|---|
ChatGPT-User |
OpenAI | Visiting a page during a question in ChatGPT or a custom GPT, GPT Actions | "robots.txt rules may not apply"; not used to decide what appears in search | https://openai.com/chatgpt-user.json |
Claude-User |
Anthropic | Fetching a site when a user asks Claude a question | Anthropic states its bots honor robots.txt; blocking Claude-User may reduce your visibility in user-directed searches |
https://claude.com/crawling/bots.json |
Perplexity-User |
Perplexity | Visiting a page to answer a question and cite it | "generally ignores robots.txt rules" | https://www.perplexity.ai/perplexity-user.json |
MistralAI-User |
Mistral AI | Visiting a page to answer a user in Mistral's assistant and cite the source; no automatic crawling | Documented token; compliance not specified | https://mistral.ai/mistralai-user-ips.json |
meta-externalfetcher |
Meta | On-demand link fetching, including for agent tasks | "may bypass robots.txt rules" | No list published |
Amzn-User |
Amazon | Live information to answer Alexa requests | "may not follow all robots.txt directives" | https://developer.amazon.com/amazonbot/live-ip-addresses/ |
Google-Agent (HTTP string) |
Google-hosted agents that browse and act on request | User-triggered fetchers generally ignore robots.txt; no documented robots.txt token | https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json |
Google also lists Google-GeminiNotebook (URLs that Gemini Notebook users add as sources; the former Google-NotebookLM token is supported until August 2026) in the same category of user-triggered fetchers. Anthropic, for its part, says that browsers it hosts on a user's behalf identify as Claude-Agent.
Full published User-Agent strings
The following operators publish the exact string (Meta also documents a longer variant with a link to its documentation; Amazon and Google use W.X.Y.Z as a placeholder for the Chrome version). Version numbers can change, so match on the token, not on the whole string.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
CCBot/2.0 (https://commoncrawl.org/faq/)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
meta-externalagent/1.1
meta-externalfetcher/1.1
meta-webindexer/1.1
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
OpenAI says it may add a robots.txt marker to the string when it fetches that file. Anthropic only publishes the names (ClaudeBot, Claude-SearchBot, Claude-User), not a full string. Older tokens such as anthropic-ai or Claude-Web still appear in many lists, but they are not in Anthropic's current documentation.
Key takeaways
- A User-Agent string can be faked in one line. Only the published IP ranges (or reverse DNS for Google, Bing, Apple and Common Crawl) can authenticate a crawler. See how to verify GPTBot, ClaudeBot and PerplexityBot.
- Blocking a training crawler does not remove your site from search answers, with one exception,
Google-Extended, which also covers grounding in Gemini. - For user-initiated agents, robots.txt is not a guarantee. OpenAI, Perplexity, Meta, Amazon and Google say so explicitly; Mistral does not say either way.
- Anthropic points out that blocking its IP addresses is not a reliable way to opt out of training, because the crawler can then no longer read robots.txt.
Each crawler also has its own page in the AI crawler directory, for example GPTBot or ClaudeBot. To see how widely these tokens are actually used, the weekly AI crawler census tracks how the top 10,000 sites treat them in robots.txt and llms.txt. To check your own site, use the IABIG checker.
Official sources
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google, Google's common crawlers (Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google, User-triggered fetchers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- Apple, About Applebot: https://support.apple.com/en-us/119829
- Common Crawl, CCBot and FAQ: https://commoncrawl.org/ccbot and https://commoncrawl.org/faq
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- Meta, Meta Web Crawlers: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- Amazon, Amazonbot: https://developer.amazon.com/amazonbot
- Mistral AI, Mistral crawlers: https://docs.mistral.ai/robots
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.