Tokenizer
Software that converts content into the token units a model processes.
What Tokenizer does not show
Different tokenizers split identical text differently, so token counts are model-specific rather than properties of the page.
Where Tokenizer comes up on this site
- Text-to-HTML ratio: does it matter, and how to check itUse decompressed bytes: CSS and JSON compress very well, so compressed sizes flatter repetitive markup, and the decompressed size is what a parser walks and a tokenizer consumes.
Terms this definition uses
Related terms in Models and generation
How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
Autoregressive model · Transformer · Attention · Self-attention · Parameter · Inference · Pretraining · Fine-tuning · Supervised fine-tuning · Instruction tuning · RLHF · Preference optimization · Knowledge distillation · Quantization · Mixture of experts · Multimodal model · Vision-language model · Small language model · Temperature · Top-p sampling · Deterministic decoding · System prompt · Developer message · Tool call · Function calling · Structured output · Constrained decoding · Chain of thought · Reasoning trace · Context caching · Context compaction · Model routing · Model drift · Model version
Questions about Tokenizer
What is Tokenizer?
Software that converts content into the token units a model processes.
What does Tokenizer not show or guarantee?
Different tokenizers split identical text differently, so token counts are model-specific rather than properties of the page.
Which area of the glossary does Tokenizer belong to?
Models and generation: How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
← Small language model · Temperature →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.