Bytespider
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
What Bytespider means
The practical consequence for measurement is ordering: verify who actually fetched before attributing volume to a name, because a crawler that ignores rules is also the easiest name to forge.
Where Bytespider comes up on this site
- How to measure AI visibility yourself: three checksOperators with no published feed, such as Meta-ExternalAgent, Bytespider, SemrushBot and DataForSeoBot, are unverifiable rather than fake: report them as a separate bucket instead of folding them into either.
- How to verify Googlebot, GPTBot and other crawlers by IPBytespider and Meta-ExternalAgent are the most visible.
- How much AI crawler traffic is forged? We check the IPsMeta-ExternalAgent, Bytespider, SemrushBot and DataForSeoBot sit here; their requests can be neither confirmed nor accused.
- Linked from every page, our llms.txt was fetched 8 timesHow often each kind of URL was fetched The sixteen names counted are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Amazonbot,…
- State of AI visibility, October 2026: 139,788 homepagesGPTBot is the most blocked at 3.7%, followed by CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%).
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Bytespider
What is Bytespider?
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
What does Bytespider not show or guarantee?
ByteDance's training crawler, reported over several years by publishers and by edge providers to fetch without regard to robots.txt exclusions. It is the standing example that a robots.txt rule is a request, honoured by operators who choose to honour it and enforced by nothing.
Which area of the glossary does Bytespider belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← CCBot · agentic traffic →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.