Small models can learn from a larger model's outputs

An old idea named in 2015

Submitted 9 March 2015, Distilling the Knowledge in a Neural Network, by Geoffrey Hinton, Oriol Vinyals and Jeff Dean, proposed training a smaller model to reproduce the output distribution of a larger model or ensemble, rather than training it on labelled data alone. The method uses the larger model's soft, probability-weighted predictions as a training signal, which the paper argues carries more information than a single hard label, since a model's full ranking of runner-up answers reflects something about generalisation that a correct-or-incorrect label discards. The paper reports strong results on MNIST and a significant improvement to a production acoustic model by transferring an ensemble's knowledge into one deployable model.

The same idea in a modern report

The Phi-3 Technical Report, submitted 22 April 2024, describes phi-3-mini, a 3.8 billion parameter model small enough to be deployed on a phone, reporting 69 percent on MMLU and 8.38 on MT-Bench, scores the paper states rival much larger models such as Mixtral 8x7B and GPT-3.5. The report states its training data is a scaled-up version of the dataset used for the earlier phi-2 model, composed of heavily filtered public web data together with synthetic data, without fully specifying every source of it. Worth stating precisely: the report documents a small model trained partly on synthetic data and scoring well on stated benchmarks, not a verified claim that those numbers hold on tasks outside its own test set.

Where a licence enters the picture

Training a small model on a large model's outputs runs into a question the 2015 paper never had to address: whether the terms attached to the larger model's outputs allow this at all. OpenAI's terms of use, as retrieved on 16 September 2026, state that a user may not use output to develop models that compete with OpenAI, may not attempt to reverse engineer or discover the underlying components of its services, including its models and algorithms, and may not automatically or programmatically extract data or output. A distillation pipeline built on outputs from a hosted API sits against this clause in a way distillation from an openly licensed model, or one whose owner permits it, does not, which makes the teacher's source a separate question from the distillation method itself.

Questions to carry into your own evaluation

• Does the terms of use for the teacher model you are drawing outputs from actually permit training another model on those outputs?
• Does a reported benchmark score for a small, distilled model reflect your own task, or only the benchmarks stated in its technical report?
• Can you identify which parts of the training data are synthetic and which are filtered web text, given how much that mix can vary between reports?

The mechanism Hinton, Vinyals and Dean described in 2015 still describes how a small model learns from a larger one's soft predictions; what has changed since is that the larger model's outputs increasingly come with terms attached, and those terms, not the distillation method, are what can make the same technique permitted in one case and prohibited in another.