Speculative decoding sped up generation without changing outputs

A draft, then a verification, not an approximation

Autoregressive generation normally produces one token per forward pass through the full model. The speculative decoding paper, submitted in November 2022, instead uses a smaller, cheaper draft model to propose several tokens ahead, then runs the large target model once over that proposed sequence to check them in parallel. A modified rejection-sampling rule accepts or discards each proposed token so that the final output distribution is proven identical to sampling from the target model alone, token by token. A closely related paper on speculative sampling, submitted a few months later, describes the same mechanism and states it requires no retraining or architecture changes to either model.

What the paper proved and measured

The exactness claim is a mathematical property of the sampling rule, not an empirical average: the paper proves the accepted output has the same distribution as standard decoding, so speedup does not trade away output quality in the way a smaller or distilled model would. Measured on a T5-XXL model, the original paper reports a two-to-threefold wall-clock speedup compared to a standard T5X implementation. The speculative sampling paper reports a two-to-2.5-fold decoding speedup on a 70-billion-parameter Chinchilla model in a distributed serving setup. Google's own retrospective on the method, published in December 2024, attributes the achievable speedup to autoregressive inference being memory-bandwidth bound rather than compute bound on current hardware, so that scoring several draft tokens in one pass costs little more than scoring one.

Where the speedup disappears

The mechanism only pays off if the draft model's guesses are frequently accepted; a draft model that diverges heavily from the target model on the task at hand forces more rejections, and the wasted draft computation then erodes or reverses the benefit. The retrospective is explicit that the technique is a response to a hardware property, not a general algorithmic speedup, so its benefit shrinks on workloads or hardware where compute, not memory bandwidth, is the binding constraint. None of the three sources reports a speedup figure for models or tasks outside the ones they tested, so a quoted multiplier from one setting should not be assumed to transfer to a different model pair or serving environment.

• How closely does the draft model's output distribution match the target model on your actual traffic, not a benchmark set?
• Is the serving environment memory-bandwidth bound, where the technique helps, or compute bound, where it may not?
• Does the reported speedup include the cost of running the draft model at all, or only the accepted-token count?

The distributional guarantee is proven in the paper; the size of any speedup is an empirical property of the draft-target pairing and the hardware, and needs measuring on the specific pairing in use.