Bytespider
A crawler attributed to ByteDance that shows up widely in logs and blocklists. No official documentation of its purpose, robots.txt handling or IP addresses could be found.
| Operator | ByteDance |
|---|---|
| Purpose | Training — Not officially documented |
| robots.txt token | Bytespider |
| User-Agent | Full string not published by the operator |
| Obeys robots.txt | Not stated by the publisher |
| Official IP ranges | Not published |
| Reverse DNS | Not documented |
| Official documentation | — |
Live activity of Bytespider
No visit from Bytespider recorded on iabig.com over the last 30 days.
Last updated: 2026-10-25 12:07:01 UTC. See the Observatory
What Bytespider does
The Bytespider token appears in the logs of many websites and in most AI crawler blocklists. It is attributed to ByteDance, the parent company of TikTok.
When this entry was checked on 25 October 2026, no official ByteDance page describing the crawler could be found or read: nothing on its purpose, how the data is used, whether it honors robots.txt, or which IP ranges or reverse DNS hostnames it uses. Third-party sites point to the Toutiao webmaster platform (zhanzhang.toutiao.com), but its content could not be read or verified.
What circulates about Bytespider therefore comes from secondary sources, which often disagree:
- some say ByteDance's documentation claims compliance with the Robots Exclusion Protocol;
- others, including press reports and bot-monitoring services, describe requests to disallowed paths;
- no official source establishes how the data is used, whether for search, recommendations or model training.
Listing Bytespider as a training crawler reflects common usage, not a statement from the company.
With no published IP ranges, a request claiming to be Bytespider cannot be authenticated, and any other scraper can put the name in its user-agent string.
Block or allow Bytespider
A Disallow rule records your refusal, but compliance is not officially confirmed. Enforcing it means blocking the user agent at the server or CDN, with no way to authenticate requests.
To block it, add this to your robots.txt:
User-agent: Bytespider
Disallow: /To allow it explicitly (copy any private rules you have):
User-agent: Bytespider
Allow: /A group naming Bytespider replaces the User-agent: * group for it. Build a complete robots.txt with the generator and check it with the tester.
Verify a Bytespider visit
Anyone can write “Bytespider” in a User-Agent. Paste the IP address and User-Agent from your logs into the IP verifier to know whether the visit is genuine.