Frontier model training compute has doubled every few months

Three eras, three doubling times

Submitted to arXiv on 11 February 2022, Compute Trends Across Three Eras of Machine Learning divides the history of machine learning compute into three periods. Before roughly 2010, the paper finds training compute for notable systems doubling on a timeline consistent with Moore's law, about every 20 months. From the early 2010s, in what the authors call the Deep Learning era, the doubling time shortens to around six months. From around 2015, a smaller set of large-scale models emerges with training runs 10 to 100 times larger again, developed by a small number of well-resourced organisations.

What is being measured, and how

The paper's compute estimates come from published details where available, and otherwise from estimating hardware type, utilisation and training duration. This is corroborated by Epoch AI's public trends page, a continuously updated dashboard maintained by a research group that tracks the underlying dataset, which currently states that training compute for frontier language models has grown at five times per year since 2020, a doubling time of about 5.2 months. Both figures describe compute as an input to a training run, not a measure of what the resulting model can do; a doubling of compute is not a doubling of capability, and the paper does not claim otherwise.

The evidence has a stated margin of error

Epoch AI's own notable models database documents how uneven the underlying evidence is. Models are flagged notable by criteria such as a state-of-the-art benchmark result, high citation counts, large-scale use, or training cost above one million dollars, and estimates are labelled by confidence: "Confident" records are described as accurate within a factor of three, "Likely" within a factor of ten, and "Speculative" within a factor of thirty. A trend line built from records with those error bars describes an uneven public record, not a precise physical measurement, and the database's own note that coverage "is limited by the amount of publicly available information" should travel with any chart drawn from it.

• Is a cited compute figure taken from a paper's own disclosure, or estimated from hardware and duration, and at what confidence level?
• Does a claimed doubling time hold across the whole period cited, or only within one of the three eras the 2022 paper distinguishes?
• What capability claim, if any, is being attached to a compute trend, and does the underlying evidence support that inference?

Compute is one input among several - data, architecture, and training method also matter - and the 2022 paper is explicit that its three eras have different drivers. A trend that held from 2015 onward describes a small set of well-funded projects; it is not a law about the field, and it does not by itself predict what the next model will cost or do.