What is Distillation?

Distillation (Knowledge Distillation) is a model compression technique where a smaller 'student' model is trained to mimic the behavior of a larger 'teacher' model, transferring knowledge to create efficient models that retain much of the original performance.

Quick Facts

Full NameKnowledge Distillation
Created2015 by Hinton et al.
SpecificationOfficial Specification

How It Works

Knowledge distillation enables deploying AI capabilities in resource-constrained environments. The technique works by training the student model on the soft probability distributions (logits) output by the teacher, rather than just hard labels. This transfers 'dark knowledge' about relationships between classes. Modern applications include distilling GPT-4 into smaller models, creating efficient inference models, and building specialized models from general-purpose teachers. Notable examples include DistilBERT and various distilled LLMs.

Key Characteristics

  • Transfers knowledge from large teacher to small student
  • Uses soft labels (probability distributions) for training
  • Preserves more knowledge than training on hard labels alone
  • Enables deployment on edge devices and mobile
  • Can combine multiple teachers for ensemble distillation
  • Temperature parameter controls softness of distributions

Common Use Cases

  1. Creating efficient models for mobile deployment
  2. Reducing inference costs while maintaining quality
  3. Building specialized models from general teachers
  4. Compressing large language models
  5. Edge AI and IoT applications

Example

loading...
Loading code...

Frequently Asked Questions

What is the difference between knowledge distillation and model pruning?

Knowledge distillation trains a smaller student model to mimic a larger teacher model's outputs, transferring learned knowledge. Model pruning removes unnecessary weights from an existing model. Distillation creates a new, smaller architecture while pruning reduces an existing one.

Why is temperature important in knowledge distillation?

Temperature softens the probability distribution from the teacher model, revealing more information about relationships between classes. Higher temperatures produce softer distributions that transfer more 'dark knowledge' about class similarities, helping the student learn better representations.

Can knowledge distillation be used with large language models?

Yes, knowledge distillation is widely used for LLMs. Examples include DistilBERT (distilled from BERT) and various distilled versions of GPT models. It enables deploying powerful language capabilities on resource-constrained devices.

What is the typical performance loss when using knowledge distillation?

Student models typically retain 90-99% of the teacher's performance while being 2-10x smaller. The exact performance depends on the compression ratio, student architecture, and quality of the distillation process.

What is ensemble distillation?

Ensemble distillation combines knowledge from multiple teacher models into a single student. This can produce students that outperform any individual teacher by capturing complementary knowledge from different models.

Related Tools

AI Websites Directory

An authoritative, comprehensive, and continuously updated AI resources directory. It covers global and domestic model providers, open-source ecosystems, research indexes and leaderboards, developer platforms, and curated tool catalogs—helping you quickly discover, compare, and choose the right AI products and references. Supports keyword search and favorites, with clear category sections and an expanding dataset for better experience.

Related Terms

Quantization

Quantization is a model compression technique that reduces the precision of neural network weights and activations from higher bit representations (like 32-bit floating point) to lower bit formats (like 8-bit or 4-bit integers), significantly decreasing model size and inference costs while maintaining acceptable accuracy.

PEFT

PEFT (Parameter-Efficient Fine-Tuning) is a family of techniques that adapt large pre-trained models to downstream tasks by training only a small subset of parameters, dramatically reducing computational requirements while maintaining competitive performance.

Inference

Inference (in machine learning) is the process of using a trained model to make predictions or generate outputs on new, unseen data, representing the deployment phase where learned patterns are applied to real-world inputs without updating model parameters.

LLM

LLM (Large Language Model) is a type of artificial intelligence model trained on massive amounts of text data to understand, generate, and manipulate human language with remarkable fluency and contextual awareness, powering applications from conversational AI to code generation.

Related Articles

Model Quantization Complete Guide [2026] - From Theory to Deployment

Deep dive into LLM quantization techniques: INT8, INT4, FP16, BF16 quantization types explained, PTQ vs QAT comparison, GPTQ, AWQ, GGUF quantization methods in practice. Includes llama.cpp and bitsandbytes code examples to help you deploy lightweight LLMs efficiently.

2026-02-21

LLM Fine-Tuning Complete Guide [2026] - From Theory to Practice

Master large language model fine-tuning techniques: Full fine-tuning, LoRA, QLoRA and other PEFT methods explained. Includes Hugging Face practical code, data preparation guide, and fine-tuning vs RAG selection strategies to help you customize your own AI model.

2026-02-21

LoRA Fine-Tuning Complete Guide [2026] - Low-Rank Adaptation Principles and Practice

Deep dive into LoRA (Low-Rank Adaptation) fine-tuning technology: mathematical principles of low-rank decomposition, key parameter configuration, QLoRA quantization optimization, and PEFT library practical code. Master the core skills of efficient LLM fine-tuning with 90% memory reduction while achieving professional-grade results.

2026-02-21