Knowledge Distillation: Training a Small Model to Copy a Big One

A small, cheap 'student' model is trained to reproduce the behavior of a large, expensive 'teacher' model - which is a big reason today's small models got so much better.

The basic idea

Knowledge distillation takes a large, well-trained "teacher" model and uses it to train a much smaller "student" model. Instead of training only on raw labeled examples, you train the student to mimic the teacher's actual outputs - including its full probability distribution over next tokens, not just its single best answer.

Why that's more informative than plain labels

A plain label just says "the correct answer is X." A teacher's full distribution says something richer: "X is very likely, Y is plausible, Z is basically ruled out." That extra signal - sometimes called "dark knowledge" - teaches the student to generalize, not just memorize one right answer per example.

Why small models got so good

Many of today's small, cheap models owe their quality to distillation from a larger sibling, rather than training from scratch alone. DeepSeek's distilled models, for instance, are small dense models trained on outputs from a much larger reasoning model - and they outperform similarly-sized models trained conventionally.

What gets lost along the way

A distilled student inherits its teacher's typical behavior well but often loses ground on rare edge cases the teacher handled via broader knowledge the smaller model never absorbed. Distilled models can also be less robust to unusual phrasing, since they learned a compressed approximation, not the full capability.

EXAMPLE

Simplified illustration - a teacher's next-token distribution for "The capital of France is ___": Hard label (traditional training): "Paris" - 100%, everything else 0%. Teacher's soft distribution (what distillation actually trains on): Paris: 92% Lyon: 3% a: 2% located: 1% (all other tokens): 2% Training on the soft distribution teaches the student that "Lyon" is a plausible-but-wrong French city, while "the" or "pizza" are basically never right in this slot - information a single hard label can't convey at all. Real-world naming pattern to recognize: model names like "DeepSeek-R1-Distill-Qwen-32B" tell you directly that this is a Qwen-32B-sized student, distilled from the larger DeepSeek-R1 teacher's reasoning traces.

🛠️ EXERCISE — TRY IT YOURSELF

Compare a distilled small model against its larger teacher (or a similarly-sized non-distilled model) on the same task.

Pick a small, clearly-labeled distilled model (e.g. one of the DeepSeek-R1-Distill family) and its teacher or a same-size alternative.
Write 3-4 test questions that include at least one unusual or edge-case phrasing.
Run all questions through both models and save both sets of answers.
Compare quality on the straightforward questions vs. the edge-case ones.
Note where the small model matched the larger one, and where it clearly fell behind.

✅ SELF-CHECK

☐ Did the distilled model perform close to its teacher on the straightforward questions?
☐ Did the gap widen specifically on the unusual or edge-case phrasing?

QUICK QUIZ

What does a student model learn from during knowledge distillation that a normal 'hard label' training example doesn't provide?