A shared harness makes an evaluation score reproducible

A shared benchmark is not a shared result

Two projects have shaped how open language models get evaluated in practice: EleutherAI's lm-evaluation-harness, which its own README describes as covering more than 60 standard academic benchmarks and hundreds of task variants, and OpenAI's Evals framework, which lets a contributor define a new evaluation either through a template and a data file or through custom scoring code. Both exist because naming a benchmark, such as MMLU or GSM8K, does not specify how a model is prompted, how its answer is extracted, or how that answer is scored, and two research groups implementing the same benchmark from its paper alone can and do build different pipelines.

Why the same task name gives different scores

The lm-evaluation-harness README sets out an explicit priority order for resolving that ambiguity: follow the convention a model's own trainers used if one is widely agreed, then an official implementation, then whatever evaluators have converged on informally, before falling back on any of several common but inconsistent approaches. The maintainers' own retrospective, Lessons from the Trenches on Reproducible Evaluation of Language Models, names the resulting problem directly, pointing to the sensitivity of models to evaluation setup and a general lack of reproducibility and transparency as recurring obstacles the paper tries to codify into shared guidance. A harness fixes this by pinning the prompt template, the answer-extraction rule and the scoring function in code that two people can run and get the same number from, rather than leaving each detail to be reconstructed from a paper's methods section.

What running your own task still requires

Using a harness does not remove judgement from evaluation, it relocates it. Choosing lm-evaluation-harness's configured task format or writing a new template-based eval in OpenAI's Evals still requires deciding what counts as a correct answer, how partial credit is handled, and whether the model is shown few-shot examples or none. A score produced by a harness is comparable to another score from the same harness version and configuration; it is not automatically comparable to a number quoted from a paper that used a different one, however similar the benchmark's name. The tool guarantees that a result can be reproduced, not that it means the same thing everywhere it is quoted.

Questions to carry into your own evaluation

• Which harness and which version produced the number you are comparing against?
• Was the model tested with the same prompting convention, such as zero-shot or few-shot, in both cases?
• Would writing the task yourself surface a scoring choice the original number left unstated?

A benchmark name promises a task. A harness is what makes the score attached to that name mean the same thing twice.