A leaderboard reset when its own benchmarks saturated

A leaderboard built on borrowed benchmarks

Hugging Face's Open LLM Leaderboard ranked open-weight models on a fixed set of academic benchmarks, and by mid-2023 was already prominent enough that a Hugging Face post from 23 June 2023 used it to investigate a specific problem: the same MMLU benchmark, run through three different implementations, produced different scores and even reordered which model looked best. The post reports LLaMA-65B scoring 0.636 under the original Berkeley implementation, 0.637 under Stanford's HELM, and 0.488 under the EleutherAI harness version current at the time, a gap the authors say means the mere indication of an MMLU result gives little information about whether two published numbers are actually comparable.

Why the board was rebuilt

On 26 June 2024, the leaderboard team published a post titled Open-LLM performances are plateauing, let's make the leaderboard steep again, retiring all six original benchmarks: ARC, HellaSwag, the original MMLU, Winogrande, GSM8K and TruthfulQA. The stated reasons were saturation, since models were reaching baseline human performance on several of them; contamination, meaning models showing signs of having trained on benchmark data or close equivalents; and dataset errors, citing independent investigations that found mistakes in MMLU's reference answers. Six harder replacements followed, including MMLU-Pro, GPQA and BBH. A separate, earlier post recorded a narrower version of the same integrity problem: a December 2023 deep dive into the DROP benchmark found that its scoring cut off correct floating-point answers and penalised longer, higher-quality responses, leading the team to remove it rather than leave a broken score in place.

What a rank does and does not show

The June 2024 post states plainly that different evaluation results are not always correlated with one another, which is itself a claim about the leaderboard's design: an aggregate rank averages across benchmarks that can measure different things, so a model ranked first overall is not established as best at the specific task a reader cares about. A leaderboard position is evidence that a model scored well on the particular tasks currently included, under the particular scoring code currently used, at the particular moment the board last recomputed its numbers, none of which is guaranteed to hold for a task outside that set.

Questions to carry into your own evaluation

• Do any of the leaderboard's included benchmarks resemble the task you actually need done?
• Has the model's training data plausibly overlapped with the benchmark it is ranked on?
• Would the rank survive being recomputed by a different harness implementation?

A public leaderboard is useful exactly to the extent that its benchmarks still measure something models have not already absorbed. Hugging Face's own posts are a record of that usefulness expiring and being rebuilt, not a guarantee that it will not expire again.