Speculative Decoding Explained: Draft-and-Verify for Faster LLMs

Waiting for a large language model to finish typing out an answer feels like watching paint dry. You know the model has the intelligence to answer instantly, but it’s stuck in a slow, step-by-step loop. This isn’t just annoying; it’s expensive. In production environments, every millisecond of delay costs money and frustrates users. The bottleneck isn’t the model’s brain-it’s how we ask it to think.

The solution lies in a technique called speculative decoding. It doesn’t make the model smarter. Instead, it makes the model faster by changing how it generates text. By using a "draft-and-verify" approach, you can speed up token generation by 2x or more without losing any quality. If you are deploying LLMs today, understanding this pipeline is no longer optional-it’s essential for keeping latency low and costs down.

Why Traditional Autoregressive Generation Is Slow

To understand why speculative decoding works, you first need to see what’s breaking in standard generation. Most modern transformers use autoregressive decoding. This means the model predicts one token at a time. It looks at the prompt, predicts the next word, adds that word to the context, and then predicts the word after that. It repeats this cycle until the sentence is complete.

This process is inherently sequential. You cannot predict the fifth word until you have predicted the fourth. But here is the catch: modern GPUs are designed for parallel processing, not sequential tasks. When a GPU waits for one token to be calculated before starting the next, most of its computing power sits idle. Furthermore, LLM inference is memory-bound, not compute-bound. The bottleneck is loading the massive model weights from VRAM into the processor, not the math itself. Since each token requires a full pass through those weights, the cost per token remains high regardless of how fast your chip is.

How the Draft-and-Verify Pipeline Works

Speculative decoding borrows a concept from computer architecture known as speculative execution. CPUs have used this for decades to guess the outcome of a branch prediction and execute instructions ahead of time. If the guess is wrong, the CPU discards the work and starts over. If it’s right, it saves time. We apply this same logic to LLMs.

The pipeline involves two models: a small, fast draft model and a large, accurate target model. Here is the three-phase loop:

1. Draft Generation: The small draft model (which might be 10x smaller than the target) quickly predicts the next K tokens (usually 3 to 10). Because it is small, it loads weights from memory much faster.
2. Parallel Verification: Instead of verifying these tokens one by one, we feed the entire sequence of draft tokens into the large target model in a single forward pass. Thanks to the transformer architecture, this single pass calculates the probability distribution for the next token at every position in the sequence simultaneously.
3. Rejection Sampling: The system compares the probabilities. For each draft token, it checks if the target model agrees. If the target model assigns a higher or equal probability to the draft token than the draft model did, the token is accepted. If the target model disagrees significantly, the token is rejected, and all subsequent draft tokens are discarded. The target model then generates the correct next token, and the cycle restarts.

This method ensures that the final output is statistically identical to what the large model would have produced alone. You get the speed of the small model with the quality of the large one.

The Math Behind Rejection Sampling

You don’t need to implement the math yourself if you use libraries like vLLM, but understanding it helps you debug performance issues. The core mechanism is rejection sampling. Let’s look at a concrete example.

Imagine the draft model proposes the sequence: "discovered a breakthrough."

• Token 1: "discovered". The draft model assigns a probability $P(draft) = 0.6$. The target model assigns $P(target) = 0.8$. Since $0.8 \ge 0.6$, the token is accepted.
• Token 2: "a". The draft model assigns $P(draft) = 0.7$. The target model assigns $P(target) = 0.75$. Since $0.75 \ge 0.7$, this token is also accepted.
• Token 3: "breakthrough". The draft model is confident, assigning $P(draft) = 0.5$. However, the target model thinks this is unlikely, assigning $P(target) = 0.2$. Since $0.2 < 0.5$, there is a chance of rejection. The algorithm accepts this token with probability $P(target)/P(draft)$, which is $0.2/0.5 = 0.4$. If it is rejected (which happens 60% of the time), "breakthrough" and any following drafts are thrown out. The target model then generates its own preferred token, perhaps "new," and the speculation restarts from there.

In the best-case scenario, where all K draft tokens are accepted, the system generates K+1 tokens in a single iteration. The target model verifies K tokens and produces one additional token beyond them. With K=5, you get six tokens for the price of one forward pass. That is a 6x theoretical speedup, though real-world results typically hover around 2x to 3x due to occasional rejections.

Medusa Architecture: A Faster Alternative

While traditional speculative decoding uses two separate models, newer architectures optimize this further. One notable innovation is the Medusa architecture, introduced in research from 2024. Instead of running a separate draft model, Medusa adds multiple prediction heads directly onto the base LLM.

These heads are simple feed-forward layers attached to the last hidden layer of the transformer. They allow the model to predict the next K tokens in a single inference pass without waiting for the previous token to be generated sequentially. This creates a tree of possible continuations. Medusa eliminates the overhead of switching between two different model weights, reducing memory traffic even further. It is particularly effective when you want to accelerate a specific model without maintaining a separate draft model ecosystem.

Implementing Speculative Decoding in Production

You do not need to build this from scratch. Major inference frameworks have integrated support for speculative decoding. If you are using vLLM, you can enable it with configuration flags. vLLM optimizes this specifically for medium-to-low query-per-second (QPS) workloads where memory bandwidth is the constraint. It handles the complex logic of draft verification and rejection sampling automatically.

For those using the Hugging Face Transformers library, the feature is often referred to as "assisted generation." You simply pass the draft model alongside the target model during initialization. A common pairing in 2026 is using a lightweight model like Gemma2-2B-it as the drafter for a larger Gemma2-9B-it verifier. This combination has shown significant latency reductions in chatbot applications.

When choosing a draft model, aim for one that shares the same tokenizer and architectural family as the target model. Mismatched tokenizers can cause alignment issues, while vastly different architectures may lead to lower acceptance rates. The goal is to find a draft model that is fast enough to save memory load time but accurate enough to keep the acceptance rate above 60-70%.

Edge Cases and Limitations

Speculative decoding is not a magic bullet. Its effectiveness depends heavily on the acceptance rate. If the draft model is too weak, the target model will reject most tokens, adding computational overhead without gaining speed. In such cases, standard autoregressive generation might actually be faster.

Additionally, this technique shines in long-context scenarios. If your prompts are short, the overhead of managing the draft pipeline may outweigh the benefits. It is also less effective for highly creative or unpredictable outputs where the target model’s probability distributions are flat (high entropy). In these cases, the draft model struggles to predict correctly, leading to frequent rejections.

However, for structured tasks like code generation, translation, or summarization-where the next tokens are highly predictable-speculative decoding delivers consistent 2x to 3x speedups. As hardware evolves and models grow larger, the memory-bound nature of inference will only intensify, making techniques like draft-and-verify critical for sustainable AI deployment.