token
The unit a model reads, roughly a word fragment.
What token means
Token counts matter because markup, navigation and boilerplate consume the same budget as the content they surround.
Where token comes up on this site
- AI crawler list: what each bot does and how to verify itCrawlCheck’s catalogue holds 114 named crawler identities , each admitted only when the operator’s own documentation names the token.
- XML sitemap audit: check the file and every URL in itVerify the actual response, then tell it apart from a wall Fetch the headers and the body separately and read both: curl -sS -D /tmp/sitemap.headers -o /tmp/sitemap.xml https://example.com/sitemap.xml cat /tmp/sitemap.headers head -c 500 /tmp/sitemap.xml…
- How to test text color contrast on hover and keyboard focusOne token, four surfaces, one pass.
- HSTS check: missing header, bad preload, removed entriesA header that sends the preload token without meeting the preload list’s requirements: .
- CSP report-only: how to roll out a Content-Security-Policy without breaking your siteIf report-to appears, confirm its value is a bare token and that a Reporting-Endpoints header defines that token.
Terms this definition uses
Definitions that name token
Chunk boundary · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · Autoregressive model · Tokenizer · Top-p sampling · Deterministic decoding · Constrained decoding · Fixed-size chunking · Late interaction · Robots group merging · HSTS preload list · protected-resource metadata · token audience · PKCE
Related terms in Answer engines and retrieval
How a generated answer gets made, and what your page is competing for inside it.
AEO · GEO · AI visibility score · share of voice · prompt set · brand mention · RAG · grounding · training data · embedding · chunking · context window · hallucination · refusal · AI Overview · answer engine · citation · retrieval · hallucinated attribution · Hybrid search · Re-ranking · Passage ranking · Query fan-out · Chunk boundary · BM25 · cross-encoder · AI Mode · information gain · dense retrieval
Questions about token
What is token?
The unit a model reads, roughly a word fragment.
Which area of the glossary does token belong to?
Answer engines and retrieval: How a generated answer gets made, and what your page is competing for inside it.
← context window · hallucination →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.