A million-token context window passed a retrieval test

A context window measured in millions, not thousands

On 15 February 2024, Google announced Gemini 1.5 Pro in a post titled Our next-generation model: Gemini 1.5, describing a standard 128,000-token context window with an experimental mode extending to 1 million tokens for early testers, and stating that researchers had tested versions up to 10 million tokens internally. The scale mattered because most deployed models at the time worked in the tens of thousands of tokens; a context window in the millions promised feeding in an entire codebase, a lengthy recording's transcript, or hundreds of documents at once, rather than chunking and retrieving fragments piece by piece.

How near-perfect recall was measured

The Gemini 1.5 technical report documents this claim with a needle-in-a-haystack (NIAH) methodology: a short, specific fact is inserted at a chosen position inside a long filler text, and the model is asked to retrieve it. The report states 'near-perfect retrieval (>99%)' up to at least 10 million tokens on text, and describes similar results when the needle is embedded in video or audio instead. This is a test of finding one planted fact in a haystack of otherwise irrelevant material. It is not a test of drawing a conclusion that depends on combining several facts scattered across the input, and it is not a test of noticing when no answer exists in the haystack at all.

What retrieval accuracy does not establish

The RULER benchmark paper, submitted two months later, argues directly that NIAH is 'a superficial form of long-context understanding' and introduces harder variants: multiple needles, needles that must be aggregated rather than simply located, and multi-hop tracing across the input. Testing a range of contemporaneous long-context models against these variants, the RULER authors found that only about half maintained what they judged satisfactory performance even at 32,000 tokens, well short of the millions advertised, with accuracy dropping further as inputs grew longer. Neither Google's report nor the RULER paper claims Gemini 1.5 reasons reliably across its full advertised window; the report is silent on multi-hop aggregation tasks, and RULER evaluates a broad set of models rather than issuing a verdict on any single one. The distinction that matters for a practitioner is between a model finding a known string and a model reasoning over unstructured information distributed through a long document.

• Does the retrieval test resemble the actual task, or does the task require combining facts from separate parts of the input?
• At what context length, specifically, does the vendor's own report stop reporting numbers?
• Would a harder long-context benchmark such as RULER change the practical context length a workflow should budget for?

A published context window is a ceiling on what a model can accept, not a guarantee of what it can use. The gap between the two is exactly where a retrieval-style test and a genuine long-context task diverge, and it is worth checking against the specific class of document a workflow will actually feed the model.