Speculative Decoding with Compressed Draft Models for LLMs

Imagine waiting three seconds for a chatbot to reply when it could have answered in one. That delay isn't just annoying; it kills user engagement and burns server cash. The culprit? Autoregressive generation. Large Language Models (LLMs) spit out tokens one by one, sequentially. It’s like reading a book word-by-word instead of scanning the page. Speculative Decoding is an inference acceleration technique that uses a smaller, faster draft model to predict multiple tokens ahead, which are then verified in parallel by the larger target model. This method can slash latency by up to 3x without sacrificing output quality. But here's the catch: it only works if your draft model is actually good at guessing what the big model will say.

Why Sequential Generation Is Slow

Standard LLM inference is bottlenecked by memory bandwidth, not compute power. When you ask an LLM to generate text, it processes the input once but then generates each subsequent token based on all previous ones. This sequential dependency means you can't easily parallelize the generation step. You wait for token A to finish before starting token B. For long outputs, this adds up quickly.

Speculative decoding breaks this chain. Instead of generating one token at a time, we let a tiny, fast model guess the next few tokens. Then, the big model checks them all at once. If the guesses are right, we skip several forward passes of the expensive model. If they're wrong, we fall back to normal generation. The key insight is that verifying multiple tokens in parallel is much cheaper than generating them sequentially.

The Core Mechanism: Draft and Verify

The process has three clear steps. First, the draft model proposes the next $k$ tokens based on the current context. This happens in a single forward pass because the draft model is small and fast. Second, the target model verifies these tokens in parallel. It computes the log-likelihood probabilities for all proposed tokens simultaneously. Third, the system accepts the longest prefix of correct predictions. Any rejected token triggers the target model to generate the correct token from that point onward, and the cycle repeats.

This approach relies heavily on the acceptance rate ($\alpha$), which measures how often the target model agrees with the draft model. High acceptance rates mean fewer rejections and more speedup. Research from BentoML shows that theoretical speedups can reach 3x when acceptance rates are high. However, real-world results often vary because generic draft models don't always match the target model's distribution well.

Draft Model Architectures: Beyond Separate Models

Not all speculative decoding setups are created equal. The traditional method uses a separate, smaller model as the drafter. But newer architectures try to do better by integrating drafting directly into the target model or using smarter feature reuse.

Medusa takes a different route. Introduced in a 2024 ICML paper, it adds multiple prediction heads directly on top of the last hidden layer of the base LLM. These heads predict the next $k$ tokens in a single pass, creating a tree of possible continuations. Since there's no separate draft model, you avoid distribution mismatches and simplify deployment. Medusa has shown impressive tokens-per-second gains, especially when combined with tree-based verification strategies.

EAGLE focuses on feature reuse. Instead of predicting raw tokens, it trains a lightweight module to predict the next feature vector from the target model's penultimate layer. By leveraging rich intermediate representations, EAGLE achieves significantly better acceleration than vanilla speculative sampling. It bridges the gap between separate draft models and integrated approaches like Medusa.

The Importance of Distribution Alignment

Here’s where most implementations fail: assuming any small model works as a drafter. Your draft model must mimic the probability distribution of the target model closely. If the draft model thinks "the" is likely but the target model prefers "a," you get rejections. Each rejection wastes computation.

Industry practitioners recommend training custom draft models on domain-specific data. Generic models trained on broad internet text might perform poorly on specialized tasks like legal document summarization or code generation. BentoML’s research highlights that while off-the-shelf models can offer decent speedups, custom-trained drafters consistently achieve higher acceptance rates. If your workload is niche, don't skip the fine-tuning step.

However, if your current setup already yields high acceptance rates, additional training might be overkill. Measure first. Use tools to log acceptance rates during inference. If you’re seeing frequent rejections, invest in better drafting. If not, stick with what works.

Quality Preservation and Statistical Guarantees

One common worry: does speeding up change the output? In theory, speculative decoding preserves the exact same probability distribution as standard autoregressive decoding. This means the generated samples come from the identical statistical space. You don't sacrifice accuracy for speed.

This guarantee holds strictly under specific conditions, particularly when using speculative sampling methods grounded in rigorous math (like those extended by DeepMind). For greedy decoding-where you always pick the most probable token-the logic is even simpler. If the draft matches the argmax, accept it. If not, reject and regenerate. No complex sampling corrections needed.

In practice, slight numerical differences might occur due to floating-point precision, but these are negligible for most applications. The key takeaway: you can deploy speculative decoding in production environments where output fidelity matters, such as customer support bots or medical advice systems.

Implementation Checklist for Production

Ready to implement? Here’s what you need to consider:

• Select the Right Architecture: Do you want simplicity (Vanilla), integration (Medusa), or feature reuse (EAGLE)? Evaluate your engineering resources.
• Measure Acceptance Rates: Before optimizing, baseline your current performance. Track how many draft tokens are accepted per batch.
• Domain-Specific Training: Fine-tune your draft model on representative data from your use case. General-purpose models often underperform in specialized domains.
• Optimize Batch Size: Speculative decoding interacts with batching. Larger batches may dilute the benefits if draft models aren't optimized for concurrent requests.
• Monitor Latency vs. Throughput: Speculative decoding primarily reduces Inter-Token Latency (ITL). Check if your application cares more about response time or total throughput.

Future Directions and Optimization

The field moves fast. New variants like LayerSkip and MTP (Medusa Tree Parallel) aim to squeeze out more performance. Researchers are exploring dynamic drafting lengths-adjusting $k$ based on confidence scores rather than fixing it. Some experiments suggest adaptive strategies can further improve efficiency by avoiding unnecessary drafts when the model is uncertain.

Another trend is combining speculative decoding with quantization. Running compressed draft models alongside quantized target models amplifies speed gains. As hardware becomes more efficient, the focus shifts from pure compute to memory bandwidth optimization, making speculative techniques increasingly relevant.