IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

AI Crawler User Agents List 2026 - GPTBot, ClaudeBot, PerplexityBot and robots.txt Tokens

GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and more, with operator, robots.txt token, purpose, compliance and official IP ranges.

AI companies no longer send a single crawler. They run several agents with distinct jobs: collecting pages for model training, indexing for a conversational search engine, or fetching a page on the spot because a user asked a question. Mixing them up leads you to block your visibility when you meant to block training, or the other way around. This list only covers agents described in their operator's official documentation, last checked on November 6, 2026. Links are collected at the end of the page.

Three families to tell apart

  • Training: the crawler collects pages that may be used to train models. Blocking it should, in principle, have no effect on whether you appear in search answers.
  • Search and indexing: the crawler feeds the index that lets an assistant cite and link to your site. Blocking it reduces your visibility.
  • User-initiated actions: the agent visits a page because a user asked it to (a link pasted into the chat, a live search, an agent task). Several operators state that robots.txt may not apply to these requests.

Two special cases: Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. No HTTP request ever carries these names. The crawling is done by Googlebot and Applebot, and the token only expresses a preference about how the data is used.

Training crawlers

robots.txt token Operator Stated purpose Honors robots.txt (per operator) Published IP ranges
GPTBot OpenAI Content that may be used to train OpenAI's foundation models Yes https://openai.com/gptbot.json
ClaudeBot Anthropic Content that may contribute to model training Yes, including Crawl-delay https://claude.com/crawling/bots.json (one list shared by all Anthropic crawlers)
CCBot Common Crawl Open web archive that anyone can reuse Yes (robots.txt read before each site, Crawl-delay honored) https://index.commoncrawl.org/ccbot.json
meta-externalagent Meta Use cases "such as training foundation AI models or improving products by indexing content directly" Meta says to add a Disallow to block it No list published on Meta's page
MistralAI-Training Mistral AI Building training datasets Yes, Disallow supported No list published
Amazonbot Amazon Improving Amazon products; "may be used to train Amazon AI models" Yes, but not Crawl-delay https://developer.amazon.com/amazonbot/ip-addresses/ (HTML page)
Bytespider ByteDance Not confirmed, no official documentation found Not confirmed Not confirmed

Common Crawl does not present CCBot as a training crawler. It builds a public archive that anyone can download, including to train models. Blocking it is therefore the only way to keep your pages out of future archives.

As for Bytespider, the token shows up in every log file and blocklist, but we found no ByteDance page describing its purpose, its robots.txt behavior or its IP addresses. Claims about it come from third-party sources.

Control tokens with no crawler behind them

robots.txt token Operator What it controls Effect on search
Google-Extended Google Whether content crawled by Google is used to train future Gemini models (Gemini Apps, Vertex AI API for Gemini) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI According to Google, no effect on inclusion or ranking in Google Search
Applebot-Extended Apple Whether data crawled by Applebot is used to train Apple's foundation models According to Apple, pages still appear in search results and the rule is not used for ranking

A point that is often missed: disallowing Google-Extended also removes your content from grounding in the Gemini apps, not just from training. On Apple's side, opting out of generative answers in Siri and Search is done with the nosnippet meta tag, not with Applebot-Extended.

Search and indexing crawlers

robots.txt token Operator Stated purpose Honors robots.txt (per operator) Published IP ranges
OAI-SearchBot OpenAI Surfacing websites in ChatGPT's search features Yes; an excluded site can still appear as a navigational link https://openai.com/searchbot.json
Claude-SearchBot Anthropic Indexing the web to improve the relevance of search answers Yes https://claude.com/crawling/bots.json
PerplexityBot Perplexity Surfacing and linking websites in Perplexity search results; not used for foundation models Yes (documented robots.txt token; changes can take up to 24 hours) https://www.perplexity.ai/perplexitybot.json
Applebot Apple Search in Spotlight, Siri and Safari; data may also be used for training unless you opt out with Applebot-Extended Yes; follows Googlebot rules if Applebot is not named; ignores Crawl-delay https://search.developer.apple.com/applebot.json
Amzn-SearchBot Amazon Amazon search experiences, including Alexa; no generative training Yes; follows the rules given to other search bots if it is not named https://developer.amazon.com/amazonbot/searchbot-ip-addresses/
meta-webindexer Meta Improving Meta AI search results Meta encourages allowing it in order to be cited; compliance not specified No list published
MistralAI-Index Mistral AI Indexing for Mistral's search; no training Not specified by Mistral https://mistral.ai/mistralai-index-ips.json
DuckAssistBot DuckDuckGo Real-time fetching for DuckDuckGo's AI-assisted answers; not used to train AI models Yes; DuckDuckGo says a Disallow takes effect after 72 hours and does not affect organic rankings https://duckduckgo.com/duckassistbot.json

OpenAI notes that if both GPTBot and OAI-SearchBot are allowed, it may use the results of a single crawl for both purposes, so you will see fewer visits from one of the two in your logs.

Agents acting on behalf of a user

Token Operator Stated purpose Honors robots.txt (per operator) Published IP ranges
ChatGPT-User OpenAI Visiting a page during a question in ChatGPT or a custom GPT, GPT Actions "robots.txt rules may not apply"; not used to decide what appears in search https://openai.com/chatgpt-user.json
Claude-User Anthropic Fetching a site when a user asks Claude a question Anthropic states its bots honor robots.txt; blocking Claude-User may reduce your visibility in user-directed searches https://claude.com/crawling/bots.json
Perplexity-User Perplexity Visiting a page to answer a question and cite it "generally ignores robots.txt rules" https://www.perplexity.ai/perplexity-user.json
MistralAI-User Mistral AI Visiting a page to answer a user in Mistral's assistant and cite the source; no automatic crawling Documented token; compliance not specified https://mistral.ai/mistralai-user-ips.json
meta-externalfetcher Meta On-demand link fetching, including for agent tasks "may bypass robots.txt rules" No list published
Amzn-User Amazon Live information to answer Alexa requests "may not follow all robots.txt directives" https://developer.amazon.com/amazonbot/live-ip-addresses/
Google-Agent (HTTP string) Google Google-hosted agents that browse and act on request User-triggered fetchers generally ignore robots.txt; no documented robots.txt token https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json

Google also lists Google-GeminiNotebook (URLs that Gemini Notebook users add as sources; the former Google-NotebookLM token is supported until August 2026) in the same category of user-triggered fetchers. Anthropic, for its part, says that browsers it hosts on a user's behalf identify as Claude-Agent.

Full published User-Agent strings

The following operators publish the exact string (Meta also documents a longer variant with a link to its documentation; Amazon and Google use W.X.Y.Z as a placeholder for the Chrome version). Version numbers can change, so match on the token, not on the whole string.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
CCBot/2.0 (https://commoncrawl.org/faq/)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)
meta-externalagent/1.1
meta-externalfetcher/1.1
meta-webindexer/1.1
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

OpenAI says it may add a robots.txt marker to the string when it fetches that file. Anthropic only publishes the names (ClaudeBot, Claude-SearchBot, Claude-User), not a full string. Older tokens such as anthropic-ai or Claude-Web still appear in many lists, but they are not in Anthropic's current documentation.

Key takeaways

  • A User-Agent string can be faked in one line. Only the published IP ranges (or reverse DNS for Google, Bing, Apple and Common Crawl) can authenticate a crawler. See how to verify GPTBot, ClaudeBot and PerplexityBot.
  • Blocking a training crawler does not remove your site from search answers, with one exception, Google-Extended, which also covers grounding in Gemini.
  • For user-initiated agents, robots.txt is not a guarantee. OpenAI, Perplexity, Meta, Amazon and Google say so explicitly; Mistral does not say either way.
  • Anthropic points out that blocking its IP addresses is not a reliable way to opt out of training, because the crawler can then no longer read robots.txt.

Each crawler also has its own page in the AI crawler directory, for example GPTBot or ClaudeBot. To see how widely these tokens are actually used, the weekly AI crawler census tracks how the top 10,000 sites treat them in robots.txt and llms.txt. To check your own site, use the IABIG checker.

Official sources

Free tools

Related guides