Benchmark scores can measure memorisation, not skill

The problem: test data inside training data

A benchmark score is informative only if it measures a model's ability to solve new problems, not its ability to recall problems it has already seen. Several studies published between late 2023 and early 2024 examine how often that assumption fails in practice. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, submitted 8 November 2023, shows that the common defence, checking for exact string overlap between training data and test questions, misses paraphrased duplicates: a 13-billion-parameter model trained on rephrased benchmark questions could match GPT-4-level scores on benchmarks including MMLU, GSM8K and HumanEval, despite never seeing the exact wording. It proposes an LLM-based decontamination check that looks for semantic rather than surface overlap, and reports that popular pretraining corpora such as RedPajama and StarCoder-Data already contained 8 to 18 percent overlap with HumanEval.

How contamination has been found in deployed models

Leak, Cheat, Repeat, submitted 6 February 2024, takes a different approach: rather than testing a controlled model, it examines how research practice itself leaks benchmarks into closed models such as GPT-3.5 and GPT-4. Analysing 255 papers, it estimates that these models were collectively exposed to roughly 4.7 million samples from 263 benchmarks through published research use in their first year of availability, and separately catalogues evaluation malpractices such as missing baselines and reproducibility gaps. A third paper, Task Contamination, submitted 26 December 2023, uses a chronological method: it compares model performance on datasets released before versus after a model's training cutoff and finds models perform better on the older, potentially-seen data, while showing little advantage over a majority-class baseline on tasks where contamination is structurally impossible.

What this does and does not establish

Together, these are three different detection strategies converging on the same conclusion: contamination is present in specific, documented cases, not merely a theoretical risk. None of the three establishes a universal contamination rate for any current model, and each method has its own limit: rephrase-based detection depends on the judge model's own semantic sensitivity, citation counting cannot see private fine-tuning data, and chronological comparison cannot rule out that newer benchmarks are simply harder. A reader should treat a high score on an old, public benchmark as a claim that needs a decontamination check, not as false by default.

Questions to carry into your own evaluation

• Was the benchmark's exact text, or a close paraphrase of it, publicly available before the model's training cutoff?
• Did the model developer publish a decontamination method, and does it check for paraphrase, not only exact-string overlap?
• Is there a comparable, more recent or private benchmark where the model's score holds up?

The practical upshot across all three papers is the same: a benchmark score is a claim about a specific dataset at a specific point in time, and its evidentiary value declines the longer that dataset has been public.