Decoding settings shape output as much as the prompt does

A named failure mode in generated text

Submitted on 22 April 2019, The Curious Case of Neural Text Degeneration documented that decoding strategy, not model quality alone, shapes the character of generated text. The authors found that maximising likelihood at each step, as greedy decoding and beam search do, produces text that is bland and strangely repetitive, even from a model that performs well on understanding tasks. They also reported systematic distributional differences between human-written and machine-generated text under likelihood-maximising decoding, evidence that the decoding rule itself, applied on top of an unchanged model, is enough to produce or avoid degeneration.

Truncating the tail instead of always taking the peak

The paper's proposed fix, nucleus or top-p sampling, samples from the smallest set of next-token candidates whose cumulative probability exceeds a chosen threshold, discarding the long, unreliable tail of unlikely tokens rather than sampling from the entire distribution or always taking the single most likely token. This preserves some randomness, which the paper found necessary for varied, human-like text, while cutting off exactly the low-probability options most associated with incoherence. OpenAI's current API reference, documented as retrieved on 16 September 2026, describes its own top_p parameter in similar terms: a value of 0.1 means only tokens comprising the top ten percent of probability mass are considered, and the documentation recommends altering top_p or temperature but not both at once.

A different knob, doing a different job

Temperature rescales the probability distribution before sampling, making the model more deterministic near zero and flatter, more random, near its upper bound; it does not by itself cut off an unreliable tail the way nucleus sampling does. The 2019 paper treats temperature and truncation as separate levers, and current vendor documentation still separates them: Anthropic's Messages API reference, current as retrieved on 16 September 2026, describes temperature as the amount of randomness injected into a response, on a 0.0 to 1.0 scale, and notes that even a temperature of 0.0 does not guarantee fully deterministic output. The same page states that temperature has been withdrawn as a settable parameter on some newer model releases, a reminder that a decoding setting documented today is not guaranteed to exist on tomorrow's model.

Questions to carry into your own evaluation

• Are you changing temperature and top_p together, against the documented guidance to alter one and not both?
• Does your task need the exploratory variety nucleus sampling preserves, or the narrower output a low temperature is meant to produce?
• Does a temperature of 0.0 actually make your outputs repeatable on the model you are using, or only closer to it?

A prompt sets what the model is asked to do; the decoding settings sitting beside it set how the model turns its own probability distribution into the next token, and the 2019 paper's distinction between the two remains a more reliable frame than treating sampling parameters as a minor afterthought to prompt wording.