Constrained decoding made JSON output reliable, not correct

A schema the model cannot miss

On 6 August 2024, OpenAI announced Structured Outputs, a mode in which, according to the announcement, a model's response will reliably produce valid JSON matching a supplied schema once a developer sets strict mode on. The company reported that on an internal evaluation of complex schema-following, the gpt-4o-2024-08-06 model reached 100% reliability with the feature enabled, against under 40% for gpt-4-0613 without it. That is a comparison between one model with the feature and an older model without it, run on OpenAI's own evaluation, not a general claim about every schema or every model.

How the guarantee is enforced

The mechanism behind this kind of guarantee is constrained decoding: at each generation step, the set of tokens the model is allowed to produce next is restricted to those consistent with the target grammar, so the output cannot leave the shape of the schema. Willard and Louf's paper on efficient guided generation describes a version of this approach that reformulates a schema or regular expression as a finite-state machine and builds an index over the model's vocabulary so that only valid next tokens are considered, adding little overhead to generation. OpenAI's own guide confirms the practical effect: once the constraint is active, a developer does not need to worry about the model omitting a required key or producing an invalid enum value, because the decoder will not produce a token that breaks either rule.

Valid JSON is not correct JSON

A schema constrains shape, not content. The same guide lists what the guarantee does not cover: a refusal for safety reasons, a response cut off by a token limit, or an incompatibility with parallel function calls will all still occur, and none of OpenAI's documented conditions catch an arithmetic mistake or a wrong value placed correctly inside a valid field. A response can satisfy every type and enum constraint in a schema while still containing a wrong total, a fabricated identifier, or a mismatched unit, because the schema was never designed to check any of those things. Structured Outputs solves a parsing problem, not a correctness problem, and the two are easy to conflate once a pipeline stops throwing exceptions on malformed JSON.

Questions to carry into your own evaluation

• Does the schema cover every field your downstream code assumes is present, or only some of them?
• What happens to a request when a refusal or a token limit prevents a schema-conforming response?
• Is anything checking the values inside a valid response, not just their shape?

A parser that stops throwing exceptions is a real improvement. It is not the same achievement as a model that stops being wrong.