CCBot
The crawler of Common Crawl, a non-profit that maintains an open archive of the web that researchers, companies and individuals can download for free.
| Operator | Common Crawl |
|---|---|
| Purpose | Training — Common Crawl open web archive |
| robots.txt token | CCBot |
| User-Agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Obeys robots.txt | Yes, according to the publisher |
| Official IP ranges | https://index.commoncrawl.org/ccbot.json — 5 ranges, list dated 2026-08-11 |
| Reverse DNS | crawl.commoncrawl.org |
| Official documentation | https://commoncrawl.org/ccbot |
Live activity of CCBot
No visit from CCBot recorded on iabig.com over the last 30 days.
Last updated: 2026-10-25 14:07:01 UTC. See the Observatory
What CCBot does
CCBot is the crawler of Common Crawl, a non-profit that provides a free copy of part of the web to researchers, companies and individuals for research and analysis. The data is released as open data with as few restrictions as possible.
What Common Crawl states:
- CCBot checks robots.txt first and only fetches a page if crawling it is allowed. The token to use is
CCBot. - It obeys
Crawl-delayand backs off when a server returns HTTP 429 or 5xx responses. - It honors
nofollowon links, does not execute JavaScript and does not use cookies. - The archive is a sample of the web, not a full copy of each site.
- Other crawlers impersonate CCBot. It now runs on dedicated IP ranges with reverse DNS (hostnames under
crawl.commoncrawl.org, except over IPv6), and the ranges are also published as JSON.
Common Crawl does not present CCBot as an AI training crawler. Because anyone can reuse the archive, though, it can be used, among many other purposes, to train models by third parties that have no relationship with your site. Blocking CCBot is how you keep new pages out of future archives.
The user-agent format is stable (CCBot/2.0), but Common Crawl warns that the version number may be incremented.
Block or allow CCBot
Your new pages should stay out of future Common Crawl archives, and therefore out of the datasets third parties build from them.
To block it, add this to your robots.txt:
User-agent: CCBot
Disallow: /To allow it explicitly (copy any private rules you have):
User-agent: CCBot
Allow: /A group naming CCBot replaces the User-agent: * group for it. Build a complete robots.txt with the generator and check it with the tester.
Verify a CCBot visit
Anyone can write “CCBot” in a User-Agent. Paste the IP address and User-Agent from your logs into the IP verifier to know whether the visit is genuine.