The 2024 AI Index compiles measurements, not a single verdict

An annual compilation, not a single study

Stanford HAI published its account of the 2024 AI Index on 15 April 2024, in a summary built around thirteen charts. The underlying report, described on the project's own ongoing AI Index page as an "independent initiative" that "track[s], collate[s], distill[s], and visualize[s] data relating to artificial intelligence", is not a single experiment. It is a compilation of statistics drawn from many separate sources - vendor disclosures, academic datasets, and outside surveys - assembled once a year into one document covering models, training cost, government policy, and public opinion.

What the 2024 edition reported

The April 2024 summary states that organisations released 149 foundation models in 2023, of which 65.7 percent were described as open source, and that the United States produced 61 of those models. On cost, it cites a training-cost estimate of about $191 million for Google's Gemini Ultra and about $78 million for OpenAI's GPT-4, alongside a reminder that the original 2017 Transformer paper's compute is estimated to have cost roughly $900 - three figures on three very different scales, none independently audited by Stanford HAI itself, each attributed to a separate estimation exercise. The same summary reports 25 AI-related policies passed in the United States that year and survey findings on public nervousness about AI products that vary sharply by country.

A compilation inherits its sources' limits

Because the Index aggregates figures it did not itself generate, each number carries the assumptions of whoever produced it: a training-cost estimate is an estimate, a benchmark comparison used the setting its original authors chose, and a public-opinion figure reflects one survey's wording and sample. The report is useful for a first orientation to a fast-moving field, and less useful as a single citation for a specific claim. Citing "the AI Index" as a blanket source obscures which underlying study actually measured the number in question.

• Which underlying source produced the specific figure being cited, and what method did that source use?
• Is the comparison being drawn from the same edition and year, given that methodology and coverage change annually?
• Does the statistic describe a measured outcome or a survey of opinion, and is that distinction preserved when the figure is repeated elsewhere?

Used carefully, the Index is a map of what has been measured about AI in a given year, pointing toward primary sources rather than replacing them. Used loosely, as a single number to end an argument, it lends compiled estimates a precision that no one figure in it was built to carry on its own.