Batch APIs traded turnaround time for half-price inference

Half price for work that can wait

OpenAI's dated changelog records the Batch API as released on 15 April 2024. Anthropic followed six months later with its Message Batches API, announced on 8 October 2024. Both offer the same basic trade: a developer submits many requests at once instead of one at a time, accepts that results will not arrive immediately, and pays half the standard per-token price in exchange.

What each specification requires

OpenAI's Batch guide documents a 50% discount against synchronous pricing, a firm 24-hour completion window after which an incomplete batch expires, a JSONL input file capped at 200MB, and a limit of 50,000 requests per batch across chat, embeddings, moderation and several other endpoints. Anthropic's original announcement specified the same 50% discount and 24-hour window, with an initial limit of 10,000 queries per batch across Claude 3.5 Sonnet, Claude 3 Opus and Claude 3 Haiku. Its batch-processing guide adds a detail neither original announcement states outright: most batches finish in under an hour, well inside the 24-hour ceiling both vendors quote, because the window is a guarantee against a worst case rather than a typical wait.

Deciding what belongs in a batch

The discount is not free of a design constraint: a batch request is submitted and its results collected later through a separate retrieval step, so a task that depends on the previous turn's output, such as an interactive conversation or an agent loop that decides its next tool call from the last result, cannot be batched without restructuring the workflow around that delay. What batches well is large-volume, independent work: classifying a fixed set of documents, translating a corpus, or scoring many rows against the same rubric, where every request can be defined in advance and no request needs to see another's answer. The saving is real only if the 24-hour window fits the task; a batch that must return before a person is waiting on it defeats the purpose of the discount.

Questions to carry into your own evaluation

• Does any request in your batch depend on the output of another request in the same batch?
• Can your workflow tolerate a wait of up to 24 hours, even though most batches finish sooner?
• Is the volume large enough that a 50% saving offsets the added complexity of submitting and polling a batch?

Batch pricing is a scheduling decision translated into a discount. It rewards work that was never going to need an immediate answer in the first place.