Knowledge Distillation Explained: Teacher-Student Models
Key Insights
- Knowledge distillation trains a smaller student model to mimic a larger teacher model’s behaviour without directly copying its parameters.
- Soft targets preserve information about alternative predictions, while temperature controls the smoothness of the teacher’s probability distribution.
- A knowledge distillation loss function often combines teacher-output matching with the student’s task loss, using techniques such as KL divergence and cross-entropy.
- Response-based, feature-based and relation-based distillation transfer different kinds of information from the teacher to the student.
- LLM distillation can use output probabilities, teacher-generated responses, internal representations or reasoning examples, but it does not guarantee that every teacher capability will transfer to the student.
- Distillation can reduce model size and deployment requirements, but its impact on accuracy, inference speed and cost depends on the model, training data and distillation objective.
Knowledge distillation is a machine learning technique that trains a small, compact model (the student) to mimic the behaviour of a large, complex model (the teacher). The main goal of knowledge distillation is to create a more efficient model that retains as much of the teacher’s performance as possible while reducing deployment and inference costs.
Rather than copying the teacher’s parameters directly, knowledge distillation transfers learned information through predictions, internal representations or relationships. This guide explains how the teacher-student model works, how soft targets and temperature guide learning, and how distillation loss is calculated. It also explores the main types of knowledge distillation, its applications in large language models (LLMs), and its benefits and limitations.
How does the Teacher-Student Setup Work?
A teacher-student model uses a trained teacher to supervise a smaller student. The teacher provides predictions or representations, and the student learns to approximate selected aspects of that guidance.
What happens during knowledge distillation?
The typical process involves five steps:
- Select a trained teacher model with strong performance on the target task.
- Choose a smaller student model with lower deployment requirements.
- Pass training inputs through both models.
- Compare the student’s predictions or representations with the teacher’s using a distillation objective.
- Update the student’s parameters to reduce the loss, while usually keeping the teacher frozen.
For example, a large image classifier might assign probabilities to several object categories. A smaller student learns from these predictions alongside, or instead of, the original labels.
Must the teacher and student have the same architecture?
No. The student model vs teacher model comparison is primarily about their roles, not identical architectures. A teacher may be a large Transformer, while the student uses fewer layers or hidden dimensions. However, matching their outputs or intermediate features may require projection layers or other alignment mechanisms.
Boundary: The teacher can offer more information than a single correct answer. Soft targets and temperature explain how that information becomes a useful training signal.
What are Soft Targets and Temperature in Knowledge Distillation?
Soft targets are probability distributions produced by the teacher rather than single hard labels. Temperature controls how smooth these distributions become, helping the student learn relationships between competing predictions.
How are soft targets different from hard labels?
Suppose a teacher classifies an image containing a cat. A hard label identifies the answer simply as “cat.”
A possible soft-target distribution is:
- Cat: 0.70
- Dog: 0.25
- Car: 0.05
These illustrative probabilities show that the teacher considers a dog more plausible than a car. This relative information can help the student learn similarities that a single hard label does not express.
How does temperature affect the output distribution?
The temperature in knowledge distillation adjusts the sharpness of the teacher’s predicted probabilities. The calculation is:
pᵢ = softmax(zᵢ / T)
Here, zᵢ is the logit for class i, and T is the temperature. A higher temperature generally produces a softer distribution, while a lower temperature makes it sharper. At T = 1, the usual softmax distribution is retained.
Temperature is a training parameter in this context. It should not be confused with the sampling temperature that controls randomness when generating text at inference time.
Boundary: Soft targets describe what the student should learn to imitate. A loss function measures how closely its predictions or representations match that guidance.
What is the Loss Function Used in Knowledge Distillation?
The knowledge distillation loss function measures the difference between the teacher’s guidance and the student’s output. A common approach combines a distillation loss, often based on KL divergence, with a task loss calculated against ground-truth labels.
How is distillation loss calculated?
A common combined objective is:
L = αT²L_KD + (1 − α)L_task
Where:
- L_KD is the distillation loss between the teacher’s and student’s softened output distributions.
- L_task measures the student’s error against ground-truth labels, often using cross-entropy for classification.
- T is the temperature used to soften the distributions.
- α controls the relative contribution of the two losses, with 0 ≤ α ≤ 1.
KL divergence measures how different one probability distribution is from another. In response-based distillation, it can encourage the student’s softened predictions to approximate those of the teacher. The T² factor is commonly used to compensate for changes in gradient scale caused by temperature.
This is a common formulation, not a universal objective. Some methods use only distillation loss, while others add feature-matching or relational objectives.
Boundary: The loss depends on what the student is expected to learn. This leads to three common categories of knowledge distillation.
What are the Main Types of Knowledge Distillation?
Response-based, feature-based and relation-based distillation differ in whether the student learns from the teacher’s final outputs, internal representations or relationships among representations.
Response-based distillation
This method transfers the teacher’s final predictions or logits. Soft targets and KL-divergence loss are common choices when both models produce comparable output distributions.
Feature-based distillation
Feature-based distillation encourages the student to reproduce selected intermediate representations from the teacher. When their dimensions differ, projection layers may help align the features before calculating the loss.
Relation-based distillation
Relation-based distillation focuses on relationships among samples, features, tokens or layers rather than matching individual outputs alone. These relationships can convey structural patterns learned by the teacher.
The categories can overlap: a training method may combine output matching with feature or relation matching.
Boundary: These methods describe how knowledge is transferred across neural networks. In language models, the same idea can help train smaller models for more efficient deployment.
How is Knowledge Distillation Used in LLMs and Small Language Models?
Knowledge distillation in LLMs trains a smaller language model to imitate selected behaviours of a larger teacher. Depending on the method, the student may learn from next-token distributions, teacher-generated responses, internal representations or reasoning examples.
How does DistilBERT demonstrate knowledge distillation?
DistilBERT is a smaller version of BERT developed using knowledge distillation during pretraining. Its training combines language-modelling, distillation and cosine-distance losses.
The original DistilBERT paper reported a model that was 40% smaller, retained 97% of BERT’s language-understanding capabilities, and was 60% faster. These are reported results from that specific study, not guarantees for every distilled model or deployment setting.
DistilBERT illustrates how distillation can create a more compact language model while retaining much of the original model’s capability.
How is distillation applied to modern LLMs and reasoning models?
LLM distillation techniques can use teacher-generated examples, token-probability distributions or selected reasoning traces as training signals. In reasoning-model distillation, a smaller model may learn from examples of how a teacher approaches multi-step problems.
However, reproducing selected outputs or reasoning traces does not guarantee that the student acquires all of the teacher’s reasoning abilities. Outcomes depend on the quality of the examples, the training objective and the student’s capacity.
What are the benefits and limitations of distillation?
The benefits and limitations of knowledge distillation depend on the model and deployment task.
Potential benefits
- Lower inference costs and memory requirements.
- Reduced deployment footprint.
- Potentially lower latency on suitable hardware.
Limitations
- The student may lose some capabilities of the teacher.
- Performance depends on training data, model capacity and the distillation objective.
- Speed and accuracy gains must be measured on the intended workload.
Distillation can complement pruning, which removes selected model components, and quantization, which represents weights or activations at lower numerical precision. These are different techniques and can also be used together.
Boundary: Knowledge distillation is useful when a smaller model meets the required quality and deployment constraints. Evaluation should establish whether the trade-off works for the intended task.
Conclusion
Knowledge distillation trains a smaller student model to learn useful behaviour from a larger teacher through output distributions, internal features or relationships. Soft targets, temperature and suitable loss functions guide this process, but a smaller model does not automatically reproduce every capability of its teacher. Its value lies in balancing task performance with deployment efficiency.
To build a stronger understanding of Generative AI, Transformer architectures and efficient language models, explore the Generative AI programme by IIT Delhi, delivered on Varsity by InterviewBit.
Frequently Asked Questions
Is knowledge distillation the same as model compression?
No. Knowledge distillation is one technique for creating a smaller or more efficient model. Model compression is the broader goal and also includes methods such as pruning and quantization.
Does the teacher model change during distillation?
In standard teacher-student distillation, the teacher is usually frozen while the student is updated. Some specialised methods use different training arrangements.
Can a student model outperform its teacher?
Yes, on a particular task or evaluation, but this is not guaranteed. The result depends on the student architecture, training data, objective and evaluation setup.
Does knowledge distillation require labelled data?
Not always. Depending on the method, distillation can use labelled datasets, unlabelled inputs, teacher-generated responses or other sources of teacher supervision.
Why is temperature scaling important during distillation?
Temperature controls the sharpness of probability distributions. Softer targets can reveal relationships among alternative predictions that hard labels do not capture.
What is the difference between distillation loss and cross-entropy loss?
Distillation loss measures how closely the student matches the teacher’s guidance. Cross-entropy commonly measures the student’s predictions against ground-truth labels. Many approaches combine both.
Can knowledge distillation work with models of different sizes?
Yes. A smaller student can learn from a larger teacher, although differences in output dimensions or internal representations may require alignment mechanisms.
Is knowledge distillation useful for edge devices?
It can be useful because a smaller student may require less memory and computation. Suitability depends on the model, hardware, latency requirements and task performance.
How do you evaluate whether distillation worked?
Compare the distilled student with the teacher and an appropriate non-distilled student baseline on the same evaluation data. Measure task quality alongside latency, memory use and inference cost.
Can knowledge distillation be combined with quantization?
Yes. Distillation trains the student using teacher guidance, while quantization reduces the numerical precision used to represent weights or activations. The combined model should be evaluated for both quality and efficiency.
Resources
- Distilling the Knowledge in a Neural Network
- Knowledge Distillation Tutorial — PyTorch
- DistilBERT: A Distilled Version of BERT
- DistilBERT Documentation — Hugging Face Transformers
- Knowledge Distillation of Large Language Models — Microsoft Research
- Knowledge Distillation for Language Models — ACL Anthology
