What is Distillation?
Distillation (Knowledge Distillation) is a training method in which a student model learns from signals produced by a teacher model, such as probability distributions, logits, intermediate representations, relationships, or generated responses, often to build a smaller or cheaper model.
Quick Facts
| Full Name | Knowledge Distillation |
|---|---|
| Created | 2015 by Hinton et al. |
| Specification | Official Specification |
How It Works
Classical logit distillation
For teacher and student logits z_t and z_s, temperature T produces softened distributions with softmax(z/T). A common objective combines hard-label cross-entropy with T^2 * KL(p_teacher || p_student). The scale factor compensates for temperature's effect on gradient magnitude. Temperature and loss weights are tunable; no fixed values work across tasks or capacity gaps.
The transferred signal defines the variant
Response distillation matches final outputs, feature distillation aligns intermediate representations, and relation distillation preserves pairwise or structural information. For generative models, teams may match token distributions, train on teacher-generated sequences, or combine both. Training only on generated text is often called black-box or sequence distillation, but it does not expose the teacher's full probability distribution.
A stronger teacher does not guarantee a stronger student
Teacher mistakes, biases, calibration errors, and policy failures can be inherited. A severe capacity or tokenizer mismatch can make the target hard to represent, while narrow transfer data can omit important behavior. Compare against the unmodified student and a same-budget supervised baseline; otherwise an apparent gain cannot be attributed to distillation.
Evaluate the exact deployment artifact
Measure target-task quality, calibration, safety and subgroup regressions, then benchmark model bytes, peak memory, warm and cold latency, throughput, and cost on target hardware. Distillation does not guarantee a compression ratio or fidelity level. Record teacher and student revisions, data provenance, generation settings, licenses, and whether teacher outputs may legally be used for training.
Key Characteristics
- Uses a teacher's outputs or representations as an additional training signal
- Often combines a distillation objective with ground-truth supervision
- Can transfer logits, features, relationships, tokens, sequences, or multiple signals
- Usually optimizes a quality and resource trade-off rather than exact replication
- Can inherit teacher errors and lose behavior outside transfer-data coverage
- Requires baselines and artifact-level quality, latency, memory, and policy evaluation
Common Use Cases
- Training a smaller classifier from a high-capacity ensemble or teacher
- Reducing inference latency or memory while preserving required task quality
- Transferring selected language-model behavior into a deployable student
- Combining pruning or quantization with recovery training
- Creating task-specific students under explicit data and licensing controls
Example
Loading code...Frequently Asked Questions
How is knowledge distillation different from pruning or quantization?
Distillation trains a student to imitate selected teacher signals. Pruning removes parameters or structures from a model, while quantization changes numerical precision. They can be combined, but each has a different failure mode and must be evaluated against the same quality and resource targets.
Why does knowledge distillation use temperature?
Dividing logits by a temperature above one creates a softer probability distribution that exposes relative preferences beyond the top class. The same temperature is applied to teacher and student in the distillation term. Too much smoothing can erase useful distinctions, so temperature must be validated rather than copied from a recipe.
Is training on teacher-generated answers the same as logit distillation?
No. Generated-answer training observes sampled or decoded sequences and is commonly called sequence or black-box distillation. Logit distillation matches richer token or class distributions and usually requires teacher internals or API support. The methods can be combined but provide different supervision.
How much teacher quality will a distilled model retain?
There is no universal percentage. Retention varies by task, metric, data coverage, student architecture and capacity, tokenizer compatibility, objective, and teacher behavior. Report per-task results against the teacher, the base student, and a same-budget non-distilled baseline instead of a single aggregate claim.
What should be checked before shipping a distilled model?
Freeze model, tokenizer, data, and decoding revisions; evaluate target quality, safety, calibration, and regression slices; benchmark latency, throughput, memory, and cost on target hardware; and verify data provenance, teacher-output permissions, licenses, monitoring, and rollback criteria.