Knowledge distillation
Training a smaller model to reproduce behavior from a larger teacher.
What Knowledge distillation means
It can reduce cost, but omissions and errors from the teacher can also be transferred.
Related terms in Models and generation
How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
Autoregressive model · Transformer · Attention · Self-attention · Parameter · Inference · Pretraining · Fine-tuning · Supervised fine-tuning · Instruction tuning · RLHF · Preference optimization · Quantization · Mixture of experts · Multimodal model · Vision-language model · Small language model · Tokenizer · Temperature · Top-p sampling · Deterministic decoding · System prompt · Developer message · Tool call · Function calling · Structured output · Constrained decoding · Chain of thought · Reasoning trace · Context caching · Context compaction · Model routing · Model drift · Model version
Questions about Knowledge distillation
What is Knowledge distillation?
Training a smaller model to reproduce behavior from a larger teacher.
Which area of the glossary does Knowledge distillation belong to?
Models and generation: How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
← Preference optimization · Quantization →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.