ClaudeBot
Anthropic's crawler for web content that could contribute to training its generative AI models. Anthropic says it honors robots.txt and the non-standard Crawl-delay directive.
| Operator | Anthropic |
|---|---|
| Purpose | Training — Claude models (training) |
| robots.txt token | ClaudeBot |
| User-Agent | Full string not published by the operator |
| Obeys robots.txt | Yes, according to the publisher |
| Official IP ranges | https://claude.com/crawling/bots.json — 14 ranges, list dated 2026-10-20 |
| Reverse DNS | Not documented |
| Official documentation | https://support.claude.com/en/articles/8896518 |
Live activity of ClaudeBot
Over the last 30 days, 100 visits from ClaudeBot to iabig.com: 100 IP-verified, 0 spoofed, 0 unverifiable. Last seen: 2026-10-25.
Last updated: 2026-10-25 12:07:01 UTC. See the Observatory
What ClaudeBot does
ClaudeBot collects web content that could contribute to training Anthropic's generative AI models, with the stated aim of making them more useful and safe. Restricting its access signals that the site's future content should be excluded from training datasets.
What Anthropic states:
- Its bots honor standard robots.txt directives and do not try to get around anti-bot measures such as CAPTCHAs.
- The non-standard
Crawl-delayextension is supported, for exampleCrawl-delay: 1. - Rules must be added to the robots.txt of every subdomain you want to opt out.
- Blocking Anthropic's IP addresses is not a reliable opt-out, because it stops the bot from reading robots.txt.
Anthropic publishes a single IP list shared by all three of its bots (ClaudeBot, Claude-SearchBot and Claude-User). An address on that list proves a request came from Anthropic, but not that it was ClaudeBot rather than another agent. Anthropic does not publish a full user-agent string either, only the bot names.
The older anthropic-ai and Claude-Web tokens still appear in many blocklists but are absent from Anthropic's current documentation. For questions or reports of a misbehaving bot, Anthropic asks site owners to write from an address on the domain concerned so the report can be verified.
Block or allow ClaudeBot
Your site's future content is excluded from Anthropic's training datasets. Visibility in Claude's search answers depends on Claude-SearchBot and Claude-User, which are configured separately.
To block it, add this to your robots.txt:
User-agent: ClaudeBot
Disallow: /To allow it explicitly (copy any private rules you have):
User-agent: ClaudeBot
Allow: /A group naming ClaudeBot replaces the User-agent: * group for it. Build a complete robots.txt with the generator and check it with the tester.
Verify a ClaudeBot visit
Anyone can write “ClaudeBot” in a User-Agent. Paste the IP address and User-Agent from your logs into the IP verifier to know whether the visit is genuine.
Other Anthropic crawlers
- Claude-SearchBot — Search
- Claude-User — Browsing