← All field notes

Research Watch · Model architecture · Efficiency

Distillation: teaching a small model to behave like a large one

A decade-old idea — training a small student to match a large teacher's full output distribution — is how many production-grade small models are actually built today, and curated data can matter as much as scale.

AI-assisted / research-based

This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.

Serving a frontier-scale model for every request is expensive, slow, and often unnecessary. Knowledge distillation asks a narrower, more useful question: can a much smaller model learn to approximate the behavior a larger one already learned, without repeating that learning from scratch?

A student does not need to rediscover what the teacher knows. It needs to learn what the teacher does with what it knows.

Soft targets carry more signal than the correct answer alone

Distilling the Knowledge in a Neural Network ↗, by Hinton, Vinyals, and Dean, starts from an observation about ensembles: averaging predictions across many models improves performance but is too expensive to serve at scale. Their fix trains a single smaller model — the student — not just on the ground-truth label, but on the teacher's full output distribution: the relative probability the teacher assigns to every possible answer, correct and incorrect.

Architecture / knowledge transfer

The wrong answers carry information too

FIG 01 - MOTION
Teacher soft targets training a student model A large teacher model produces a full probability distribution over possible answers for a training example, and a much smaller student model is trained to match that distribution rather than only the single correct label. TEACHERLarge model or ensemblefrozen, inference only SOFT TARGETSFull output distributionprobabilities on every class HARD LABELGround truththe one correct answer STUDENTSmall model, trainedmatches soft + hard targets wrong-answer probabilities carry information hard labels discard RESULTa student that approximates teacher behavior at a fraction of the serving cost
A teacher's relative confidence across every possible answer — not just the correct one — is the signal distillation transfers into a smaller student.

Those "soft targets" encode information a hard label discards entirely. A teacher that assigns a wrong answer even a small probability is revealing a specific, structured confusion — which categories it considers plausible near-misses. A student trained to match that full distribution absorbs some of the teacher's implicit sense of which mistakes are reasonable, not only which answer is correct. The paper reports this transferring an ensemble's accuracy gains into one deployable model, and demonstrates the same technique improving a production acoustic model at commercial scale.

The corpus is a lever independent of parameter count

Distillation is usually framed purely in terms of matching a teacher's outputs. Textbooks Are All You Need ↗, Microsoft Research's phi-1 paper, makes a related but distinct claim: a small model's training data, not only its parameter count, is a first-class design decision.

phi-1 is a 1.3-billion-parameter code model trained for four days on eight accelerators, using roughly 6 billion tokens of filtered "textbook-quality" web text combined with about 1 billion tokens of synthetically generated exercises produced with the help of a larger model. Despite its small size, the paper reports 50.6% pass@1 on HumanEval and 55.5% on MBPP — competitive coding performance for a model two orders of magnitude smaller than many contemporaries.

Pipeline / data curation

Curate the corpus before you shrink the model

FIG 02
Curated and synthetic data pipeline for a small production model Filtered textbook-quality web text and synthetically generated exercises from a larger model are combined into a small, high-quality training corpus used to train a compact model. FILTERWeb text, filtered~6B tokens, textbook-quality only GENERATESynthetic exercises~1B tokens, from a larger model CORPUS~7B curated tokenssmall, high signal-to-noise TRAIN1.3B-parameter model4 days, 8 accelerators TRADEwhat a small model is trained on can matter as much as how large it is
A small, deliberately curated corpus — part filtered web text, part model-generated exercises — can teach a compact model behavior that scale alone would otherwise have to buy.

The lesson generalizes beyond this one paper: a small model trained on a small, deliberately curated corpus can outperform expectations set by parameter count alone. Distillation and curated synthetic data are complementary levers — one transfers a teacher's decision boundary, the other controls what the student sees in the first place — and production teams increasingly use both together.

What distillation does not transfer

A student matching a teacher's output distribution is not the same as a student matching the teacher's reasoning process. Two specific gaps matter in production:

  • Inherited blind spots. If the teacher is systematically wrong or biased on a class of inputs, the student learns to reproduce that error with high fidelity, because the soft targets faithfully encode the teacher's confusion, not just its correct answers.
  • New, teacher-independent failures. A much smaller student has less representational capacity; it can develop its own failure modes on inputs the teacher handled correctly but that require capacity the student never had.

Neither of these shows up in a metric that only compares student accuracy to teacher accuracy in aggregate — both require testing the student directly against the task's actual failure modes.

What this changes in production

On-device and edge assistants

Applications with hard offline, latency, or memory constraints — mobile keyboards, embedded devices, air-gapped environments — cannot serve a frontier-scale teacher at all. A distilled student sized to the deployment target is often the only viable option, and curated training data (as in the phi-1 approach) can partially compensate for the capacity a much smaller model gives up.

High-volume, narrow-task endpoints

Classification, extraction, and routing tasks that run millions of times a day are usually well-suited to distillation: the task is narrow enough that a small student can match a large teacher's decision boundary closely, and the cost savings compound directly with volume in a way they do not for a low-traffic feature.

Latency-critical interactive paths

Autocomplete, real-time moderation, and other interfaces where response time is part of the product experience often trade a measured amount of raw capability for a student model that responds within a hard latency budget the teacher cannot meet.

A production checklist

Before shipping a distilled model, the team should be able to answer:

  • Was the student evaluated against the task's known failure modes directly, not only against aggregate agreement with the teacher?
  • If training data was partly generated by another vendor's model, does that vendor's terms of service permit using the outputs to train a competing or redistributed model?
  • Is the training corpus curated for quality and relevance, or is scale being used to compensate for a corpus that was not actually inspected?
  • Does the student's performance degrade gracefully or catastrophically on inputs outside the distillation training distribution?
  • Is there a fallback to the teacher model for requests the student's own confidence signals as uncertain?
  • What is the actual cost and latency delta being purchased, measured under realistic production concurrency rather than a synthetic benchmark?

Research referenced

Continue readingReturn to field notes →