Bytespider
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
What Bytespider means
The practical consequence for measurement is ordering: verify who actually fetched before attributing volume to a name, because a crawler that ignores rules is also the easiest name to forge.
Where Bytespider comes up on this site
- How to verify Googlebot, Bingbot and AI crawlers by IPBytespider is the most visible example.
- Which AI crawlers actually visit a small business siteBytespider arrived most, and went for a PDF ByteDance’s crawler made more visits than any other named agent, and the path it requested most was not the homepage.
- Our llms.txt is linked from every page. Crawlers fetch it about as often as a blog postHow often each kind of URL was fetched The sixteen names counted are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Amazonbot,…
- Release notes 3 Oct 2026: Resolve, census, rulebookOf the 273 reachable sites that serve a robots.txt, 25.3% close the homepage to at least one of the 12 AI agents; GPTBot is the most-named at 17.6%, CCBot 19.8%, Bytespider 18.7%, ClaudeBot and Google-Extended 14.7% each.
- 139,788 homepages in four days: what AI crawlers getGPTBot is the most blocked at 3.7%, followed by CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%).
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Bytespider
What is Bytespider?
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
What does Bytespider not show or guarantee?
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
Which area of the glossary does Bytespider belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← CCBot · agentic traffic →
See it in the full glossary · 731 terms across 19 areas.