Scaling laws made language model loss predictable

A power law fit to three factors

On 23 January 2020, researchers largely from OpenAI posted Scaling Laws for Neural Language Models, reporting that a transformer's cross-entropy loss falls as a power law in three quantities considered one at a time: the number of non-embedding parameters, the size of the training dataset in tokens, and the compute used for training. The paper states these trends hold across more than seven orders of magnitude and that other architectural choices, such as depth versus width, have only a minor effect once total parameter count is fixed.

What the numbers say to do with a compute budget

The paper's full text fits specific exponents to each relationship and uses them to work out how a fixed compute budget should be split. Its answer is uneven: optimal model size grows roughly as compute raised to the power 0.73, while the data processed and the number of training steps grow far more slowly, so that most of any increase in compute should go toward a larger model. The same fit implies that doubling model size only requires the dataset to grow by roughly five times, not eight, to avoid a measurable overfitting penalty. The paper concludes that big models may be more important than big data, and that compute-efficient training means stopping a large model well short of convergence rather than training a smaller one to completion.

What a later paper revised

Two years on, the Chinchilla paper retested the same question with a wider sweep of model and data sizes and reached a different practical recipe: model size and training tokens should scale together, doubling both rather than favouring size, and a compute-optimal 70-billion-parameter model trained on far more tokens than the 2020 recipe implied outperformed several larger contemporaries. The 2020 paper's own appendix already flagged this as a live uncertainty, stating it had no solid theoretical understanding for the scaling relations and listing untested regimes, including the small-data case and unexplored learning-rate interactions, as caveats on how far the fitted exponents should be trusted.

Questions to carry into your own evaluation

• Is a scaling claim describing model size held against data size, or against a jointly optimised allocation of a fixed compute budget?
• Which recipe, the 2020 model-favouring allocation or the 2022 token-matching one, does a given training decision actually follow?
• Does the cited exponent come from the same tokenizer, dataset and non-embedding parameter accounting the original paper used?

The 2020 scaling-laws paper established that loss falls predictably with scale and proposed a specific allocation of a compute budget that a later, wider study revised. Its numbers were superseded, not falsified, by work using a broader experimental sweep, a reminder that a scaling exponent is a fit to a particular set of runs, not a fixed law.