Caching a prompt prefix cut cost for repeated context

Reusing a processed prefix

Anthropic introduced prompt caching in a public beta in mid-August 2024, for Claude 3.5 Sonnet, Claude 3 Opus and Claude 3 Haiku; the same page carries a later note recording that the feature reached general availability on 17 December 2024. OpenAI followed with its own version, confirmed by a dated changelog entry as one of several features released at DevDay on 1 October 2024. Both let a developer avoid reprocessing a long, unchanged block of context, such as a system prompt, a codebase summary or a set of instructions, on every call. The two designs are not identical, and the difference matters more than the shared name suggests.

What each announcement specified

OpenAI's version applies automatically once a prompt exceeds 1,024 tokens, caching the longest previously seen prefix, at a 50% discount on the cached portion, with entries typically cleared after 5 to 10 minutes of inactivity and always within an hour. Anthropic's mechanism, documented in its developer guide, is deliberate rather than automatic: a developer marks an explicit breakpoint in the prompt, pays a 25% premium to write that prefix to the cache, and then reads it back at a 90% discount for a default five-minute lifetime that resets on each use. Anthropic's cache also depends on an exact match up to the breakpoint; changing a tool definition, an image, or content before the marked point produces a different cache entry rather than a partial hit.

Why the layout of a prompt now matters

Both designs reward the same habit: put the content that stays identical across calls first, and put whatever changes, such as a user's latest message or a timestamp, last. A developer who places a per-request value before a large static block loses the cache on every call under either vendor's rules, since neither system caches a prefix behind unstable content. The two discount structures are not interchangeable either; a workload with many short repeated calls suits OpenAI's automatic, no-setup caching, while a workload built around one long static context reused deliberately across a session suits Anthropic's explicit breakpoints and longer optional lifetime. Neither vendor's percentage discount is a current price, and both remain subject to change; what a developer can rely on is the mechanism each document describes.

Questions to carry into your own evaluation

• Is your prompt structured with the stable content first and the variable content last?
• Is your call pattern frequent enough, within each provider's cache lifetime, to benefit from caching at all?
• Does a small change to a tool definition or system instruction silently invalidate a cache you are relying on?

Caching did not make a model cheaper to run. It made a specific, disciplined way of writing a prompt cheaper than a careless one.