Chinchilla showed smaller models trained longer can win

A different split of the same compute budget

DeepMind researchers posted Training Compute-Optimal Large Language Models on 29 March 2022, reporting that for a fixed training compute budget, the number of training tokens should be doubled every time model size is doubled. That recipe directly revises the allocation set out in the 2020 scaling-laws paper, which had fit optimal model size to grow roughly as compute to the power 0.73 while data grew far more slowly, favouring large models trained on comparatively little data.

What the paper tested

The authors trained a family of models over a wider range of sizes and token counts than the earlier study, then trained a 70-billion-parameter model, Chinchilla, on the token count implied by the revised fit. The paper reports that Chinchilla uniformly and significantly outperforms Gopher at 280 billion parameters, GPT-3 at 175 billion, Jurassic-1 at 178 billion and Megatron-Turing NLG at 530 billion, despite having far fewer parameters than any of them. On the MMLU benchmark it reports 67.5% accuracy, more than seven percentage points above Gopher. The paper frames this as evidence that many contemporary large models were undertrained relative to their size, not undersized relative to their training data.

What the paper does not cover

The comparison is about training compute efficiency: reaching a given quality for a given training-time compute spend. The paper separately notes that Chinchilla, being smaller than the models it outperforms, also uses substantially less compute for fine-tuning and inference, but this is a stated side benefit of choosing a smaller model, not a metric its central fitting exercise optimises for. A team instead holding a fixed inference-cost budget or a fixed latency target is asking a different question than the one this paper answers, and the compute-optimal token count for training says nothing directly about the cost of serving the resulting model at scale.

Questions to carry into your own evaluation

• Is a claimed compute-optimal recipe being applied to a training budget, an inference budget, or both at once?
• Does a model comparison hold training compute roughly fixed, or compare models trained with very different total compute?
• Has a cited benchmark such as MMLU been checked for the kind of prompting and scoring variation that can shift its reported number?

Chinchilla's result was about training efficiency: more tokens and fewer parameters, for the same compute, produced a stronger model on the benchmarks tested. It said nothing about which model is cheapest to run afterward, and pairing a training-compute-optimal recipe with an inference-cost decision requires a separate calculation the paper does not provide.