Small scoring choices can swing a benchmark by points

Three years of running the same kind of test

Submitted 23 May 2024, Lessons from the Trenches on Reproducible Evaluation of Language Models was written by Stella Biderman and 29 co-authors drawing on three years of experience building and maintaining the Language Model Evaluation Harness, a framework the repository describes as testing generative models on more than 60 standard academic benchmarks and serving as the backend for the Hugging Face Open LLM Leaderboard. The paper sets out to document what the authors call the tacit or folk knowledge practitioners accumulate about evaluation but rarely write down, covering the sensitivity of models to evaluation setup and the difficulty of making comparisons across methods genuinely fair.

Setup, not just the model, decides the number

The paper's central concern is that small, easily overlooked choices in how a benchmark is run change the reported score independent of the model being tested: how a prompt is formatted, how an answer is extracted and scored, and how decoding is configured can each shift a result, which means two papers reporting different numbers for what looks like the same benchmark may not be measuring a genuine capability difference. The harness repository documents its response to that problem directly: evaluation with publicly available, fixed prompts is presented as what makes results reproducible and comparable between papers, and where multiple common implementations of a task exist without field-wide agreement, the maintainers state a rule for resolving the choice, prioritising consensus among model trainers first and a documented default afterward.

An earlier attempt at the same problem

HELM, submitted 16 November 2022, tackled a related version of the reproducibility gap by testing 30 models under identical conditions across 42 scenarios and measuring seven separate dimensions, including accuracy, robustness, fairness, and efficiency, rather than one aggregate score. HELM reports that before its own publication, models in the field had been evaluated on a shared set of core scenarios only 17.9 percent of the time, a figure describing how little like-for-like comparison existed across published results at that point, not a claim about any model's later scores. Read together, the two papers describe the same underlying problem from different ends: HELM standardises what gets measured, and the 2024 paper documents the setup choices within a single benchmark run that can still move a number even after the benchmark itself is fixed.

Questions to carry into your own evaluation

• Have you published the exact prompt template and scoring method you used, so someone else could reproduce your number?
• Are you comparing two reported scores that used different prompt formats or decoding settings under the same benchmark name?
• Does your evaluation report more than one metric, given that an aggregate accuracy figure can hide where a model actually fails?

A benchmark name on its own is not enough information to compare two results; the prompt, scoring method, and decoding settings behind the number are part of what was measured, and both papers here argue that reporting them is what makes a score worth citing at all.