ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers compared
What ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.
Anthropic runs three agents that may visit your site, each with its own robots.txt token. ClaudeBot collects content that could be used to train Anthropic's models, Claude-SearchBot indexes pages to improve Claude's search answers, and Claude-User fetches pages when a Claude user asks a question that needs them. Blocking one does not block the others, so you can refuse training and still be read and cited by Claude. This guide compares the three, explains what Anthropic says about robots.txt, gives robots.txt recipes and covers a practical limitation: Anthropic publishes a single IP list shared by all three bots. It is based on Anthropic's help center article on crawling as of November 2026.
The three bots at a glance
| ClaudeBot | Claude-SearchBot | Claude-User | |
|---|---|---|---|
| Purpose | Collecting web content that could contribute to training Anthropic's models | Indexing content to improve the relevance and accuracy of Claude's search results | Retrieving pages when a Claude user asks a question that needs them |
| Crawls automatically | Yes | Yes | No, user-initiated |
| Follows robots.txt | Yes, per Anthropic | Yes, per Anthropic | Yes, per Anthropic (no exception stated) |
| Effect of blocking | Future content excluded from training datasets | May reduce visibility and accuracy in Claude's search results | May reduce visibility in user-directed search |
| IP list | claude.com/crawling/bots.json, shared by all three |
same | same |
Directory pages: ClaudeBot, Claude-SearchBot and Claude-User.
ClaudeBot, the training crawler
ClaudeBot collects public web content that could contribute to training Anthropic's generative AI models. Anthropic says that restricting ClaudeBot's access signals that your site's future content should be excluded from its training datasets.
Note the word "future". Anthropic frames the opt-out as applying from now on and does not say anything about content collected before you added the rule.
Blocking ClaudeBot has no stated effect on Claude's search features. Those depend on Claude-SearchBot and Claude-User, which you configure separately.
Claude-SearchBot, the search crawler
Claude-SearchBot navigates the web to improve the quality of search results for Claude users. According to Anthropic, it analyzes online content specifically to make search responses more relevant and accurate. Disabling it prevents your content from being indexed for search optimization, which may reduce your site's visibility and accuracy in user search results.
If you want Claude to know about your pages ahead of time and cite them in search answers, this is the bot to allow. Anthropic's documentation does not say whether content indexed by Claude-SearchBot can also be used for training.
Claude-User, the on-demand agent
Claude-User supports Claude users. When someone asks Claude a question, Claude may access websites with this agent to answer it. Anthropic presents Claude-User as the way site owners control which sites can be accessed through these user-initiated requests, and says that disabling it prevents its system from retrieving your content in response to a user query, which may reduce your visibility for user-directed web search.
This is where Anthropic differs from several other vendors. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta says meta-externalfetcher may bypass it. Anthropic, by contrast, states that its bots honor robots.txt and lists no exception for Claude-User. That statement is made for all of Anthropic's bots together; the documentation has no section specific to user-initiated requests.
What Anthropic says about robots.txt
Anthropic's help center article makes these commitments for its bots:
- They honor robots.txt directives.
- They do not bypass CAPTCHAs or other anti-circumvention technologies.
- They support the non-standard
Crawl-delayextension "where appropriate", so you can slow them down instead of blocking them. The article's only example is for ClaudeBot and it does not say whether this applies equally to all three bots. - To opt out, add the rule to the robots.txt file in the top-level directory of your site, and do it for every subdomain you want to opt out.
blog.example.comandshop.example.comeach serve their own robots.txt. - Blocking Anthropic's IP addresses is not a reliable opt-out, because it can stop the bots from reading your robots.txt in the first place.
To report a misbehaving bot, Anthropic asks site owners to contact it from an email address on the domain concerned, so the report can be verified.
Two further points in the same article are easy to miss:
- A general "no AI training" signal. Anthropic says it is rolling out support for a machine-readable preference based on an Internet-Draft submitted to the IETF AI Preferences (aipref) working group,
draft-kelley-aipref-training-preference-signal, written by an Anthropic employee and published on 30 October 2026. The draft expresses the preference inside robots.txt, with a reserved group name:User-Agent: __usage_ai-trainingfollowed by ordinaryDisallowandAllowrules. It is an individual draft with no IETF endorsement, and Anthropic warns that its handling may change as standards evolve. Anthropic also says that a robots.txt group naming ClaudeBot takes priority over this general signal; the signal only applies when ClaudeBot is not named. - A fourth name. Browsers that Anthropic hosts to act on a user's behalf identify as
Claude-Agent, which is covered in a separate help article rather than alongside the three bots above.
Old tokens you can drop
Many blocklists still contain anthropic-ai and Claude-Web. Neither appears in Anthropic's current documentation, which lists only ClaudeBot, Claude-SearchBot and Claude-User. Keeping the old names is harmless, but they do not replace the current ones. A robots.txt that names only anthropic-ai does not express a documented preference to any current Anthropic bot.
Decision table
| Your goal | ClaudeBot | Claude-SearchBot | Claude-User |
|---|---|---|---|
| Maximum visibility in Claude | Allow | Allow | Allow |
| Refuse training, stay visible in Claude | Disallow | Allow | Allow |
| Refuse training and pre-indexing, still answer live user questions | Disallow | Disallow | Allow |
| Stay out of Claude entirely | Disallow | Disallow | Disallow |
| Reduce load without opting out | Allow with Crawl-delay |
Allow | Allow |
robots.txt recipes
Under RFC 9309, a crawler that finds a group naming it applies that group only, not the * group. Repeat any exclusions from * in the named groups.
Refuse training, stay visible in Claude
# Anthropic training: refused
User-agent: ClaudeBot
Disallow: /
# Claude search and user-initiated fetches: allowed
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /admin/
Allow: /
User-agent: *
Disallow: /admin/
Slow ClaudeBot down instead of blocking it
User-agent: ClaudeBot
Crawl-delay: 1
Crawl-delay is not part of RFC 9309. Anthropic supports it, but many other crawlers, including Googlebot, ignore it.
Stay out of Claude entirely
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /
Copy the file to every subdomain you want covered.
Telling them apart in your logs
Anthropic publishes the bot names but not full user-agent strings. In practice you identify each bot by the token it carries in the user-agent header:
awk -F'"' '{ ua = tolower($6) }
ua ~ /claude-searchbot/ { print "Claude-SearchBot"; next }
ua ~ /claude-user/ { print "Claude-User"; next }
ua ~ /claudebot/ { print "ClaudeBot" }' /var/log/nginx/access.log | sort | uniq -c
This assumes the default "combined" log format, where the user agent is the sixth field when a line is split on double quotes. Classifying each line once avoids double counting: a user-agent string can contain the bot name twice, for example in a contact address after the token.
Authentication is where Anthropic's setup differs from OpenAI's. OpenAI publishes one IP list per agent; Anthropic publishes a single list at https://claude.com/crawling/bots.json, in the common prefixes JSON format, for all three bots. An address on that list proves the request came from Anthropic, but it cannot tell you whether it was ClaudeBot, Claude-SearchBot or Claude-User. Only the user-agent token says that, and the token itself can be spoofed. No reverse DNS hostname is documented.
import ipaddress, json, urllib.request
URL = "https://claude.com/crawling/bots.json"
def from_anthropic(ip):
"""True if ip is in Anthropic's shared list. It does not identify which bot."""
req = urllib.request.Request(URL, headers={"User-Agent": "ip-check/1.0"})
with urllib.request.urlopen(req, timeout=10) as r:
prefixes = json.load(r)["prefixes"]
addr = ipaddress.ip_address(ip)
return any(
addr in ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
for p in prefixes
)
print(from_anthropic("216.73.216.10"))
For a quick manual check, use the crawler IP verification tool. Two consequences follow from the shared list:
- An allow rule on your firewall for Anthropic's ranges lets all three bots through. If you want to treat them differently, combine the IP check with the user-agent token, and keep robots.txt as the main control, since Anthropic says its bots follow it.
- A request with a ClaudeBot user agent from an address outside the list is not from Anthropic, whatever the header says.
Our guides on verifying GPTBot, ClaudeBot and PerplexityBot and on AI crawler traffic in server logs go into firewall rules and log analysis in more detail.
What Anthropic does not confirm
- A full user-agent string for any of the three bots.
- Which bot a given IP address belongs to.
- What happens to content collected before an opt-out.
- Whether
Crawl-delayapplies to Claude-SearchBot and Claude-User as well as ClaudeBot. - Whether content indexed by Claude-SearchBot or fetched by Claude-User can be used for training.
- How the
__usage_ai-trainingsignal is treated by services other than ClaudeBot during the rollout.
For OpenAI's equivalent agents, see GPTBot vs OAI-SearchBot vs ChatGPT-User. For a robots.txt covering every major AI company, see how to opt out of AI training.
Official sources
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler: https://support.claude.com/en/articles/8896518
- Anthropic, crawler IP addresses: https://claude.com/crawling/bots.json
- IETF, Machine-Readable Training Preference Signal (Internet-Draft): https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Meta, Meta Web Crawlers: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- Google Search Central, A note on unsupported rules in robots.txt: https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.