GSM8K needs several correct steps, not one

8.5K word problems with worked solutions

The Training Verifiers to Solve Math Word Problems paper, submitted to arXiv on 27 October 2021, introduced GSM8K: 8.5 thousand grade-school-level word problems, linguistically varied but requiring only elementary arithmetic. The dataset repository splits this into 7.5 thousand training and 1 thousand test problems, each given as a natural-language worked solution with calculator annotations marking every arithmetic step, ending in a final numeric answer. Problems are written to need between two and eight reasoning steps, deliberately more than a single calculation but still within reach of a middle-school student.

Why the authors proposed training a verifier

The paper's own framing is that even large transformer models of 2021 struggled with this conceptually simple problem distribution, failing not because the arithmetic was hard but because a wrong step early in a multi-step solution derails the final answer. Rather than only improving the generator, the authors trained a separate verifier model to judge candidate solutions and select the highest-ranked one from several sampled attempts. They report that this verification approach improved GSM8K performance and scaled better with additional data than fine-tuning the generator alone, an early example of using extra inference-time computation, rather than a larger single forward pass, to close a reasoning gap.

Why later scores on it are hard to interpret cleanly

A 2024 paper, A Careful Examination of Large Language Model Performance on Grade School Arithmetic, submitted 1 May 2024, tested this directly by building GSM1k, a new benchmark matched to GSM8K's difficulty, human solve rates and step count but containing none of the original questions. It found performance drops of up to 8 percentage points on GSM1k versus GSM8K for some model families, alongside a measurable correlation between a model's likelihood of reproducing a GSM8K example and the size of its performance gap. The same paper notes that frontier models showed little sign of this pattern and that models generally still solved genuinely novel problems, so the finding is evidence of partial, uneven overfitting in parts of the field rather than proof that GSM8K scores are meaningless.

Questions to carry into your own evaluation

• Was a reported multi-step reasoning score produced by direct generation, or by a verifier or voting scheme selecting among several attempts?
• Is there a comparable score on a newer, non-overlapping benchmark such as GSM1k for the same model family?
• Does the target task require the same two-to-eight-step arithmetic structure GSM8K tests, or longer chains where different failure modes may appear?

GSM8K's design, worked steps rather than a bare answer, made partial credit and verification possible; its age and public availability are exactly why a plain GSM8K score now needs a contamination-aware second look.