Speculative Decoding Is the Most Underrated LLM Inference Trick

Most LLM inference optimization conversations collapse into the same three topics: quantization, batching, and KV-cache tuning. Those are real levers. But there is a fourth technique that delivers comparable gains with a property none of the others can claim: it does not change the output distribution at all. Speculative decoding is provably lossless, can cut wall-clock latency by half or more in the right conditions, and is still absent from the majority of production deployments. That gap is worth examining.

The Core Idea: Cheap Drafts, Parallel Verification

Standard autoregressive generation is serial by construction. Each token depends on all previous tokens, so you cannot generate token n+1 until token n is committed. The model's forward pass is expensive, and you pay it once per token. For a 70B-parameter model on a batch of one, that cost is almost entirely memory-bandwidth-bound — you move tens of gigabytes of weights for each token produced. Throughput is miserable.

Speculative decoding breaks the serial dependency without cheating. The trick is to use a small, fast draft model — often 1–7B parameters — to greedily propose a short sequence of k candidate tokens. You then run the large verifier model on all k tokens in a single batched forward pass. Because the verifier processes all positions simultaneously, you pay one large-model forward pass instead of k. If the verifier agrees with the draft on all k tokens, you've produced k tokens for the cost of roughly one. If the verifier rejects a token at position j, you discard tokens from j onward and continue from the verifier's corrected token — still net positive if j > 1.

The acceptance criterion is not heuristic. It is a rejection-sampling scheme derived from the verifier's probability distribution. When the draft's probability for a token is at least as high as the verifier's, the token is always accepted. When the draft overshoots — assigns higher probability than the verifier would — tokens are accepted with probability proportional to the ratio of verifier to draft probability, and a correction sample is drawn. The result is that the sequence of accepted tokens is distributed exactly as if the verifier had generated them alone. No quality tax. No approximation.

Where the Speedup Actually Comes From

The gain is real but contingent. It depends on the acceptance rate — the average fraction of draft tokens the verifier accepts. If the draft and verifier agree most of the time, you amortize one expensive forward pass over many tokens. If they disagree constantly, you're doing extra work for nothing.

Acceptance rate is driven by how well the draft model approximates the verifier on your specific input distribution. A draft model trained on the same data with the same tokenizer, at a smaller scale, tends to produce high acceptance rates on prose generation and code completion. It tends to produce lower acceptance rates on tasks that require precise factual recall, complex arithmetic, or highly constrained structured outputs — exactly the cases where the verifier's knowledge advantage matters most.

The other critical variable is the draft length k. Longer drafts mean higher potential gain per verifier call but also more wasted compute when the first rejection comes early. In practice, k between 4 and 8 tends to be a reasonable range, though the optimal value is workload-dependent and worth profiling rather than guessing.

There is also a hardware dimension. Speculative decoding trades compute for memory bandwidth. On memory-bandwidth-constrained hardware — which describes most inference deployments — this is a good trade. On compute-bound hardware running large batches, the arithmetic shifts and the gains shrink. This is why speculative decoding is most compelling for latency-sensitive, low-batch-size scenarios: interactive chat, coding assistants, real-time agents.

The Variants Worth Knowing

The original formulation assumes a separate smaller model as the draft source. Several extensions are worth understanding because they change the operational complexity significantly.

• Self-speculative decoding uses early exit layers of the verifier itself as the draft model. No second model to deploy or version-manage. The draft quality is often lower than a purpose-trained small model, but the operational simplicity is a real advantage.
• Medusa attaches multiple parallel prediction heads to the verifier, each predicting tokens at different future positions. The heads are fine-tuned cheaply. Acceptance rates are competitive with small-model drafts for many tasks, and there is no separate model to serve.
• EAGLE and its successors use a lightweight autoregressive head that conditions on the verifier's hidden states rather than just output tokens. This captures more of the verifier's internal reasoning, improving acceptance rates on harder tasks at modest additional cost.
• Lookahead decoding is a related but distinct technique that uses n-gram caches instead of a learned draft model, trading some acceptance rate for zero additional parameters.

Each variant sits at a different point on the tradeoff between acceptance rate, deployment complexity, and fine-tuning cost. A separate small model gives the highest ceiling but requires maintaining two model versions in lockstep. Medusa heads require a fine-tuning run but collapse to a single artifact. Self-speculative methods require nothing extra but leave performance on the table.

Why Production Stacks Still Skip It

The theoretical case is strong. The practical adoption is lagging. Several reasons are worth naming honestly.

First, the draft model must stay in sync with the verifier. When you fine-tune or update the verifier, the acceptance rate of an existing draft model can degrade significantly. Teams that iterate rapidly on their base model find that speculative decoding introduces a maintenance dependency they didn't have before. Every model update potentially requires re-evaluating or retraining the draft.

Second, the gains are workload-specific. A benchmark showing 3× speedup on prose generation may not transfer to your structured-output extraction pipeline. Teams that don't profile on their actual traffic distribution often see disappointing results and conclude the technique doesn't work, when in fact they just need a better-matched draft model or a different variant.

Third, serving infrastructure has to support it. Speculative decoding requires the serving layer to manage draft generation, parallel verification, and token-level acceptance logic. Many inference frameworks added support relatively recently, and teams running older stacks or custom serving code face non-trivial integration work.

Fourth, it doesn't compose trivially with all other optimizations. Continuous batching, paged attention, and speculative decoding can coexist, but the interaction between dynamic batch sizes and draft acceptance rates requires careful implementation to avoid correctness issues or throughput regressions.

When to Actually Use It

The decision is straightforward once you frame it correctly. Speculative decoding is a strong candidate when:

1. Your workload is latency-sensitive and batch sizes are small — interactive generation, not offline batch jobs.
2. Your input distribution is stable enough that you can profile acceptance rates and trust they'll hold in production.
3. You can tolerate the draft-model maintenance dependency, or you're willing to use a head-based variant to eliminate it.
4. Your serving framework already supports it or you have the engineering capacity to implement it correctly.

It is a poor fit when you're running large batches where the verifier is already compute-bound, when your model updates frequently and you can't afford to track a draft model, or when your task distribution is heavily skewed toward the cases where draft acceptance rates collapse — precise factual Q&A, constrained JSON generation with complex schemas, mathematical reasoning chains.

The Deeper Point

Speculative decoding matters beyond its immediate throughput numbers. It is a clean demonstration that the autoregressive generation loop is not a physical law — it is an engineering default that can be restructured. The token-by-token serialization that makes LLM inference expensive is an artifact of how we deploy these models, not an intrinsic property of the computation.

That framing should make you curious about what other defaults in the inference stack are similarly contingent. The field has barely started pulling on this thread. Speculative decoding is the most mature result, but the underlying insight — that you can use cheap approximations to parallelize work that looks serial — will almost certainly produce more techniques as inference efficiency becomes a first-class research priority.

If you're running a latency-sensitive LLM workload today and you haven't seriously evaluated speculative decoding, that's the gap to close first.