Pretraining
Large-scale learning from broad data before adaptation to a particular task.
What Pretraining does not show
It creates general capabilities but cannot keep facts current after the training cutoff.
Related terms in Models and generation
How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
Autoregressive model · Transformer · Attention · Self-attention · Parameter · Inference · Fine-tuning · Supervised fine-tuning · Instruction tuning · RLHF · Preference optimization · Knowledge distillation · Quantization · Mixture of experts · Multimodal model · Vision-language model · Small language model · Tokenizer · Temperature · Top-p sampling · Deterministic decoding · System prompt · Developer message · Tool call · Function calling · Structured output · Constrained decoding · Chain of thought · Reasoning trace · Context caching · Context compaction · Model routing · Model drift · Model version
Questions about Pretraining
What is Pretraining?
Large-scale learning from broad data before adaptation to a particular task.
What does Pretraining not show or guarantee?
It creates general capabilities but cannot keep facts current after the training cutoff.
Which area of the glossary does Pretraining belong to?
Models and generation: How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
← Inference · Fine-tuning →
See it in the full glossary · 731 terms across 19 areas.