What Is Speculative Decoding?
Speculative decoding is a technique for making large language model inference faster without changing the model’s output. A small, fast draft model proposes several tokens ahead. The large target model then verifies those proposed tokens in a single forward pass. Where the draft guessed correctly, the system has produced multiple tokens for roughly the cost of generating one, and where it guessed wrong, it falls back to the large model’s own token.
The key property is that the result is identical to what the large model would have produced on its own. Speculative decoding changes the speed, not the answer.
Why It Helps
Normal autoregressive generation produces one token per forward pass of the large model, and each pass is expensive. For predictable text, a much smaller draft model can guess the next few tokens correctly a large fraction of the time. Verifying several proposed tokens at once amortizes the cost of the large model across multiple output tokens, which lowers latency and raises effective throughput on the same hardware.
The gain depends on the acceptance rate: how often the draft model’s proposals match what the target model would have generated. Well-matched draft and target models on predictable workloads see high acceptance and large speedups; harder or more surprising text sees less.
Speculative Decoding in Practice
Setting it up requires a compatible draft model and a serving engine that supports the verify step. It composes with the other levers of efficient serving, from continuous batching to KV cache management, and is one of the more effective ways to push down cost per token for high-volume single-model serving.
Saturn Cloud runs the inference engines that support speculative decoding on GPU infrastructure across providers, so teams can adopt it as a serving configuration rather than a research project, and measure its effect on latency and throughput directly.
