IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers compared

What ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.

Anthropic runs three agents that may visit your site, each with its own robots.txt token. ClaudeBot collects content that could be used to train Anthropic's models, Claude-SearchBot indexes pages to improve Claude's search answers, and Claude-User fetches pages when a Claude user asks a question that needs them. Blocking one does not block the others, so you can refuse training and still be read and cited by Claude. This guide compares the three, explains what Anthropic says about robots.txt, gives robots.txt recipes and covers a practical limitation: Anthropic publishes a single IP list shared by all three bots. It is based on Anthropic's help center article on crawling as of November 2026.

The three bots at a glance

ClaudeBot Claude-SearchBot Claude-User
Purpose Collecting web content that could contribute to training Anthropic's models Indexing content to improve the relevance and accuracy of Claude's search results Retrieving pages when a Claude user asks a question that needs them
Crawls automatically Yes Yes No, user-initiated
Follows robots.txt Yes, per Anthropic Yes, per Anthropic Yes, per Anthropic (no exception stated)
Effect of blocking Future content excluded from training datasets May reduce visibility and accuracy in Claude's search results May reduce visibility in user-directed search
IP list claude.com/crawling/bots.json, shared by all three same same

Directory pages: ClaudeBot, Claude-SearchBot and Claude-User.

ClaudeBot, the training crawler

ClaudeBot collects public web content that could contribute to training Anthropic's generative AI models. Anthropic says that restricting ClaudeBot's access signals that your site's future content should be excluded from its training datasets.

Note the word "future". Anthropic frames the opt-out as applying from now on and does not say anything about content collected before you added the rule.

Blocking ClaudeBot has no stated effect on Claude's search features. Those depend on Claude-SearchBot and Claude-User, which you configure separately.

Claude-SearchBot, the search crawler

Claude-SearchBot navigates the web to improve the quality of search results for Claude users. According to Anthropic, it analyzes online content specifically to make search responses more relevant and accurate. Disabling it prevents your content from being indexed for search optimization, which may reduce your site's visibility and accuracy in user search results.

If you want Claude to know about your pages ahead of time and cite them in search answers, this is the bot to allow. Anthropic's documentation does not say whether content indexed by Claude-SearchBot can also be used for training.

Claude-User, the on-demand agent

Claude-User supports Claude users. When someone asks Claude a question, Claude may access websites with this agent to answer it. Anthropic presents Claude-User as the way site owners control which sites can be accessed through these user-initiated requests, and says that disabling it prevents its system from retrieving your content in response to a user query, which may reduce your visibility for user-directed web search.

This is where Anthropic differs from several other vendors. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says meta-externalfetcher may bypass it. Anthropic, by contrast, states that its bots honor robots.txt and lists no exception for Claude-User. That statement is made for all of Anthropic's bots together; the documentation has no section specific to user-initiated requests.

What Anthropic says about robots.txt

Anthropic's help center article makes these commitments for its bots:

  • They honor robots.txt directives.
  • They do not bypass CAPTCHAs or other anti-circumvention technologies.
  • They support the non-standard Crawl-delay extension "where appropriate", so you can slow them down instead of blocking them. The article's only example is for ClaudeBot and it does not say whether this applies equally to all three bots.
  • To opt out, add the rule to the robots.txt file in the top-level directory of your site, and do it for every subdomain you want to opt out. blog.example.com and shop.example.com each serve their own robots.txt.
  • Blocking Anthropic's IP addresses is not a reliable opt-out, because it can stop the bots from reading your robots.txt in the first place.

To report a misbehaving bot, Anthropic asks site owners to contact it from an email address on the domain concerned, so the report can be verified.

Two further points in the same article are easy to miss:

  • A general "no AI training" signal. Anthropic says it is rolling out support for a machine-readable preference based on an Internet-Draft submitted to the IETF AI Preferences (aipref) working group, draft-kelley-aipref-training-preference-signal, written by an Anthropic employee and published on 30 October 2026. The draft expresses the preference inside robots.txt, with a reserved group name: User-Agent: __usage_ai-training followed by ordinary Disallow and Allow rules. It is an individual draft with no IETF endorsement, and Anthropic warns that its handling may change as standards evolve. Anthropic also says that a robots.txt group naming ClaudeBot takes priority over this general signal; the signal only applies when ClaudeBot is not named.
  • A fourth name. Browsers that Anthropic hosts to act on a user's behalf identify as Claude-Agent, which is covered in a separate help article rather than alongside the three bots above.

Old tokens you can drop

Many blocklists still contain anthropic-ai and Claude-Web. Neither appears in Anthropic's current documentation, which lists only ClaudeBot, Claude-SearchBot and Claude-User. Keeping the old names is harmless, but they do not replace the current ones. A robots.txt that names only anthropic-ai does not express a documented preference to any current Anthropic bot.

Decision table

Your goal ClaudeBot Claude-SearchBot Claude-User
Maximum visibility in Claude Allow Allow Allow
Refuse training, stay visible in Claude Disallow Allow Allow
Refuse training and pre-indexing, still answer live user questions Disallow Disallow Allow
Stay out of Claude entirely Disallow Disallow Disallow
Reduce load without opting out Allow with Crawl-delay Allow Allow

robots.txt recipes

Under RFC 9309, a crawler that finds a group naming it applies that group only, not the * group. Repeat any exclusions from * in the named groups.

Refuse training, stay visible in Claude

# Anthropic training: refused
User-agent: ClaudeBot
Disallow: /

# Claude search and user-initiated fetches: allowed
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /

User-agent: *
Disallow: /admin/

Slow ClaudeBot down instead of blocking it

User-agent: ClaudeBot
Crawl-delay: 1

Crawl-delay is not part of RFC 9309. Anthropic supports it, but many other crawlers, including Googlebot, ignore it.

Stay out of Claude entirely

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Copy the file to every subdomain you want covered.

Telling them apart in your logs

Anthropic publishes the bot names but not full user-agent strings. In practice you identify each bot by the token it carries in the user-agent header:

awk -F'"' '{ ua = tolower($6) }
  ua ~ /claude-searchbot/ { print "Claude-SearchBot"; next }
  ua ~ /claude-user/      { print "Claude-User"; next }
  ua ~ /claudebot/        { print "ClaudeBot" }' /var/log/nginx/access.log | sort | uniq -c

This assumes the default "combined" log format, where the user agent is the sixth field when a line is split on double quotes. Classifying each line once avoids double counting: a user-agent string can contain the bot name twice, for example in a contact address after the token.

Authentication is where Anthropic's setup differs from OpenAI's. OpenAI publishes one IP list per agent; Anthropic publishes a single list at https://claude.com/crawling/bots.json, in the common prefixes JSON format, for all three bots. An address on that list proves the request came from Anthropic, but it cannot tell you whether it was ClaudeBot, Claude-SearchBot or Claude-User. Only the user-agent token says that, and the token itself can be spoofed. No reverse DNS hostname is documented.

import ipaddress, json, urllib.request

URL = "https://claude.com/crawling/bots.json"

def from_anthropic(ip):
    """True if ip is in Anthropic's shared list. It does not identify which bot."""
    req = urllib.request.Request(URL, headers={"User-Agent": "ip-check/1.0"})
    with urllib.request.urlopen(req, timeout=10) as r:
        prefixes = json.load(r)["prefixes"]
    addr = ipaddress.ip_address(ip)
    return any(
        addr in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in prefixes
    )

print(from_anthropic("216.73.216.10"))

For a quick manual check, use the crawler IP verification tool. Two consequences follow from the shared list:

  • An allow rule on your firewall for Anthropic's ranges lets all three bots through. If you want to treat them differently, combine the IP check with the user-agent token, and keep robots.txt as the main control, since Anthropic says its bots follow it.
  • A request with a ClaudeBot user agent from an address outside the list is not from Anthropic, whatever the header says.

Our guides on verifying GPTBot, ClaudeBot and PerplexityBot and on AI crawler traffic in server logs go into firewall rules and log analysis in more detail.

What Anthropic does not confirm

  • A full user-agent string for any of the three bots.
  • Which bot a given IP address belongs to.
  • What happens to content collected before an opt-out.
  • Whether Crawl-delay applies to Claude-SearchBot and Claude-User as well as ClaudeBot.
  • Whether content indexed by Claude-SearchBot or fetched by Claude-User can be used for training.
  • How the __usage_ai-training signal is treated by services other than ClaudeBot during the rollout.

For OpenAI's equivalent agents, see GPTBot vs OAI-SearchBot vs ChatGPT-User. For a robots.txt covering every major AI company, see how to opt out of AI training.

Official sources

Free tools

Related guides