token
The unit a model reads, roughly a word fragment.
What token means
Token counts matter because markup, navigation and boilerplate consume the same budget as the content they surround.
Where token comes up on this site
- Does llms.txt actually do anything?If anything does read it, the cost per useful token is dramatically lower than fetching the pages it points at.
- The AI crawler and agent list: what each one is for, and which can be verifiedA name enters that catalogue on one condition: the operator’s own page names the token.
- How to audit an XML sitemap: status, content type, canonical URLs and stale entriesVerify the actual response, then tell it apart from a wall Fetch the headers and the body separately and read both: curl -sS -D /tmp/sitemap.headers -o /tmp/sitemap.xml https://example.com/sitemap.xml cat /tmp/sitemap.headers head -c 500 /tmp/sitemap.xml…
- How to test text color contrast on hover and keyboard focusOne token, four surfaces, one pass.
- HSTS audit: missing header, preload ineligible, removedNeither directive does anything on its own; preload in particular is a token that a separate submission reads, not an instruction a browser obeys.
Terms this definition uses
Definitions that name token
Chunk boundary · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · Autoregressive model · Tokenizer · Top-p sampling · Deterministic decoding · Constrained decoding · Fixed-size chunking · Late interaction · Robots group merging · HSTS preload list · protected-resource metadata · token audience · PKCE
Related terms in Answer engines and retrieval
How a generated answer gets made, and what your page is competing for inside it.
AEO · GEO · AI visibility score · share of voice · prompt set · brand mention · RAG · grounding · training data · embedding · chunking · context window · hallucination · refusal · AI Overview · answer engine · citation · retrieval · hallucinated attribution · Hybrid search · Re-ranking · Passage ranking · Query fan-out · Chunk boundary · BM25 · cross-encoder · AI Mode · information gain · dense retrieval
Questions about token
What is token?
The unit a model reads, roughly a word fragment.
Which area of the glossary does token belong to?
Answer engines and retrieval: How a generated answer gets made, and what your page is competing for inside it.
← context window · hallucination →
See it in the full glossary · 751 terms across 19 areas.