ByteSpider
AI training crawler operated by ByteDance
ByteDance
Operator
See operator's docs
Verification
Bytespider
robots.txt token
No
Honours robots.txt
Measured across detections recorded by WebDecoy, mostly on WebDecoy's own properties, 2025-11-14 to 2026-08-19. A small sample, not the whole web.
What ByteSpider is
ByteDance crawler collecting training data. Widely reported to ignore robots.txt.
It identifies itself with a User-Agent containing Bytespider. Its robots.txt token is Bytespider , but it is not listed as honouring robots.txt. Operator documentation: ByteDance .
Verifying it yourself
WebDecoy has no published IP list or reverse DNS hostname on file for ByteSpider. Check ByteDance's documentation for one, such as a list of addresses or signed requests. Until you can check the source address, the User-Agent is only a claim, and copying a User-Agent costs nothing.
Blocking it
You can ask in robots.txt, but it is not listed as honouring it:
User-agent: Bytespider
Disallow: /robots.txt only works on a client willing to obey it, which rules out every request that was already lying about who it is. Stopping those takes enforcement at the network or application layer, keyed on something the client cannot simply retype. That is the problem WebDecoy exists to solve.
Frequently Asked Questions
What is ByteSpider? +
ByteSpider is an ai training crawler operated by ByteDance. ByteDance crawler collecting training data. Widely reported to ignore robots.txt.
How do I verify ByteSpider? +
WebDecoy has no published IP list or reverse DNS hostname on file for ByteSpider. Check ByteDance's documentation for one; until you can check the source address, the User-Agent is only a claim.
How do I block ByteSpider? +
It is not listed as honouring robots.txt, so a Disallow rule for Bytespider may be ignored. Block it at your firewall, CDN or application instead, after verifying which requests really come from ByteDance.
Methodology
Every request recorded by WebDecoy's sensors, mostly on WebDecoy's own properties, whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure. Corpus: 81,034 detections from 17,187 distinct addresses, 2025-11-14 to 2026-08-19. Published ranges retrieved 2026-08-19. Operators revise their ranges over time, so a request logged months ago is checked against today's list; we re-ran the comparison over a recent 30-day window and the rates held.
This crawler dataset is licensed under CC BY 4.0; reuse is permitted with attribution to WebDecoy.
Related crawlers
- AbliterantBot Abliterant
- AI2Bot Allen Institute for AI
- Amazonbot Amazon
- Anthropic Anthropic
- Applebot-Extended Apple
- CCBot Common Crawl
- ChatGLM-Spider Zhipu AI
- Claude-Web Anthropic
- ClaudeBot Anthropic
- Cohere Cohere
- DeepSeek DeepSeek
- ElevenlabsCrawler ElevenLabs
See which ByteSpider requests are real
WebDecoy verifies declared crawlers against what their operators publish and identifies the ones that fail, as persistent actors rather than disposable IPs.