Claude's context window reached 100,000 tokens in 2023

Two announcements, two months apart

Anthropic's 100K context announcement, dated 11 May 2023, states that Claude's input window expanded 'from 9K to 100K tokens', which it describes as 'around 75,000 words', enough for 'hundreds of pages of material'. The post's own demonstration is a single worked example: Claude located a changed line inserted into the full text of The Great Gatsby, a roughly 72,000-token document, which the post says took 22 seconds. Two months later, the Claude 2 announcement, dated 11 July 2023, carried that context length into a generally available model and paired it with stated exam results: '76.5%' on a multiple-choice bar exam section, above the 90th percentile on the GRE reading and writing sections, '71.2%' on the Codex HumanEval coding benchmark, and '88.0%' on GSM8k grade-school maths problems.

What a longer window claims, and what it measures

A larger context window is a claim about how many tokens a model can accept as input, not a claim about how reliably it uses every part of that input once accepted. Anthropic's own long-context post reports one retrieval example rather than a systematic study of performance by the position of relevant information within the document. The distinction matters because a model can locate information near the start or end of a long prompt while still under-using material placed in the middle, which is a different property than raw window size and requires a different kind of test to observe.

The evidence that a bigger window is not used evenly

The Lost in the Middle paper, submitted 6 July 2023, tested multiple long-context language models on multi-document question answering and key-value retrieval, and reports a 'U-shaped performance curve': 'performance is often highest when relevant information occurs at the beginning or end' of the input, and degrades when the answer sits in the middle. The paper's tests were run on the models available to its authors at submission time and does not claim to have evaluated Claude 2 specifically, so it should be read as establishing a general finding about long-context models of that period rather than a measurement of any one vendor's product. It is nonetheless the direct evidence, from an opened source, that window size and even context use are separate properties.

Questions to carry into your own evaluation

• Does a stated context-window figure describe what the model accepts, or what it reliably uses across the full input?
• Where in a long prompt does the information your task depends on sit, and would moving it change the result?
• Do a vendor's exam-score claims specify the prompting format (few-shot, chain-of-thought) needed to reproduce them?

The 2023 jump to 100,000-token context windows was a real change in what could be submitted to a model in a single prompt. It was not, on the evidence available, a demonstrated change in how evenly a model attends to everything inside that window, and the two properties should be tested separately rather than assumed to move together.