What is speculative decoding in LLM inference?

Speculative decoding is a technique that uses a smaller, faster draft model to propose multiple tokens at once, which a larger target model then verifies in a single forward pass. When the draft predictions match the target model’s distribution, you get multiple tokens for the cost of one large-model evaluation. When they diverge, the target model corrects the trajectory and generation continues.

This approach reduces wall-clock latency for autoregressive generation without changing the target model’s output distribution — assuming the draft model is well-calibrated and the acceptance criteria are correct.

How speculative decoding works

The core loop has three stages: draft, verify, accept-or-correct. A draft model (often 10-100x smaller than the target) generates k candidate tokens autoregressively. The target model then evaluates all k+1 positions (the original context plus each draft token) in one parallel forward pass, producing its own probability distribution at each position. An acceptance criterion — typically a modified rejection sampling step — decides which draft tokens to keep and where to resample from the target distribution.

# Simplified speculative decoding loop
def speculative_generate(target_model, draft_model, prompt, max_new_tokens, gamma=4):
    """
    target_model: large model (e.g., Llama-3-70B)
    draft_model: small model (e.g., Llama-3-8B or a distilled variant)
    gamma: number of tokens to speculate per iteration
    """
    tokens = prompt
    while len(tokens) < max_new_tokens:
        # 1. Draft phase: generate gamma tokens with small model
        draft_tokens = []
        draft_logits = []
        for _ in range(gamma):
            logits = draft_model(tokens)
            next_token = sample(logits)
            draft_tokens.append(next_token)
            draft_logits.append(logits)
            tokens.append(next_token)  # speculative append
        
        # 2. Verify phase: single forward pass on target model
        target_logits = target_model(tokens[:-gamma])  # original context
        # Actually need logits at each speculated position:
        target_logits_full = target_model(tokens)  # shape: [seq_len, vocab]
        
        # 3. Accept/reject using rejection sampling
        accepted = 0
        for i, draft_tok in enumerate(draft_tokens):
            pos = len(prompt) + accepted + i
            p_target = softmax(target_logits_full[pos])
            p_draft = softmax(draft_logits[i])
            
            # Acceptance probability: min(1, p_target / p_draft)
            alpha = min(1.0, p_target[draft_tok] / p_draft[draft_tok])
            if random() < alpha:
                accepted += 1
            else:
                # Resample from corrected distribution
                corrected = p_target - p_draft
                corrected = clamp(corrected, min=0)
                corrected = corrected / corrected.sum()
                tokens[pos] = sample(corrected)
                break  # restart speculation from this position
        
        # Trim rejected tokens and continue
        tokens = tokens[:len(prompt) + accepted]
    
    return tokens

The verification step is the key insight: the target model processes the entire speculated sequence in parallel because the draft tokens are already known. This converts k sequential large-model forward passes into one, amortizing the heavy compute over multiple tokens.

Why it matters for production inference

Latency in autoregressive generation is dominated by memory bandwidth, not compute. Each target-model forward pass reads the full model weights from VRAM/HBM. Speculative decoding reduces the number of weight reads per generated token by a factor approaching the average acceptance length. If the draft model accepts 3 tokens per iteration on average, you read the 70B model weights once every 4 tokens instead of once per token.

This translates directly to higher throughput on the same hardware. A single H100 serving Llama-3-70B might achieve ~50 tokens/sec with vanilla decoding. With a well-tuned 8B draft model and gamma=4, the same hardware can push 150-200 tokens/sec for the same output quality — because the output distribution is mathematically identical to the target model alone.

The draft model itself adds negligible overhead. An 8B model fits in a fraction of the memory and its forward passes are bandwidth-bound at much lower latency. On the same GPU, draft inference often takes <5% of the target model’s time per token.

Concrete example: Llama-3 with a distilled draft

Consider serving Llama-3-70B-Instruct. You have two practical draft options:

1. Llama-3-8B-Instruct — same architecture, same tokenizer, no extra training. Acceptance rates of 60-75% on typical chat workloads.
2. A distilled 8B draft — trained via logit matching on the 70B teacher’s outputs. Can push acceptance to 80-85% but requires a training run.

# Example vLLM configuration for speculative decoding
# (vLLM 0.6+ supports this natively)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --speculative-model meta-llama/Llama-3-8B-Instruct \
  --num-speculative-tokens 4 \
  --speculative-draft-tensor-parallel-size 1 \
  --tensor-parallel-size 4

With this setup on 4xH100 (80GB), the 70B model runs TP=4. The 8B draft runs on a single GPU (or even CPU with some frameworks). The --num-speculative-tokens parameter is gamma. Typical production values range from 3-8; higher gamma increases potential speedup but lowers acceptance rate per iteration because draft errors compound.

A realistic trace for a 500-token completion:

The draft model’s 560 passes are cheap — the 8B model at TP=1 does ~4x the throughput of the 70B at TP=4. Total GPU-seconds drop by roughly 3x.

Common misconceptions

Misconception: “Speculative decoding changes the model’s outputs.”
Correct implementation preserves the target model’s exact distribution. The rejection sampling step corrects any draft bias. The only divergence comes from floating-point non-determinism (e.g., different kernel ordering on the target model when processing speculated vs. non-speculated sequences), which is negligible in practice.

Misconception: “You need a specialized draft model.”
Any smaller model sharing the tokenizer works. Same-architecture models (Llama-3-8B for Llama-3-70B) work well out of the box. Cross-architecture drafts (e.g., a tiny RNN or Mamba model) can work but require careful calibration of the acceptance criterion because their probability distributions differ more sharply.

Misconception: “Higher gamma is always better.”
Acceptance rate decays exponentially with gamma. At gamma=8, you might accept only 1.5 tokens/iteration on average — worse than gamma=4 at 3 tokens/iteration. The optimal gamma depends on the draft model’s per-token agreement rate with the target. Profile your workload.

Misconception: “It only helps with large models.”
Speculative decoding helps whenever the target model is memory-bandwidth bound. Even a 7B model can benefit from a 1B draft on the same hardware, though the absolute speedup is smaller because the 7B model already fits in cache more easily.

Misconception: “The draft model must be faster than the target per token.”
What matters is that k draft tokens + 1 target verification is faster than k target tokens. Since the target verification is one parallel pass, the draft just needs to generate k tokens in less time than the target takes for one forward pass. This is almost always true for reasonable size ratios.

When not to use it

Speculative decoding adds system complexity: two models to load, two tokenizers to align (or verify identical), extra memory for the draft model’s KV cache during speculation, and more complex scheduling logic. Avoid it when:

• Batch size is high — at large batch sizes, the target model becomes compute-bound rather than memory-bound, and the draft model’s GPU memory footprint reduces max batch size.
• Latency budget is extremely tight — the draft phase adds sequential latency before the first target verification. For very short completions (<20 tokens), the overhead may not pay off.
• Draft model quality is poor — if acceptance rate falls below ~1.5 tokens/iteration, you’re adding overhead for minimal gain. This happens with domain mismatch (e.g., code-specialized draft for general chat) or cross-architecture pairs without calibration.

Integration notes for inference gateways

If you operate a multi-model gateway, speculative decoding fits naturally as a per-request routing option. The gateway can:

1. Expose a speculative parameter in the OpenAI-compatible completion request, accepting a draft model identifier or auto.
2. Maintain draft model pools alongside target models — smaller models are cheaper to keep warm.
3. Meter usage per token generated, not per forward pass. The client pays for output tokens; the gateway internalizes the compute savings.
4. Forward cache-control hints from the target model’s KV cache layer so speculated tokens don’t pollute prefix caches incorrectly.

// Example request with speculative decoding directive
{
  "model": "llama-3-70b-instruct",
  "messages": [...],
  "max_tokens": 500,
  "speculative": {
    "draft_model": "llama-3-8b-instruct",
    "num_speculative_tokens": 4
  }
}

The gateway handles model colocation, KV cache management for both models, and the accept/reject loop — invisible to the caller. This is where an inference gateway like n4n.ai reduces operational burden: the same endpoint that handles fallback and routing also manages speculative decoding as a first-class optimization, not a custom integration per model pair.

Summary

Speculative decoding is a free lunch for LLM latency — same outputs, fewer heavy forward passes. The technique is mature, supported in vLLM, TGI, and TensorRT-LLM, and works with off-the-shelf model pairs. The main decisions are draft model selection, gamma tuning, and whether your serving stack’s batch sizes and latency targets make the trade-off worthwhile. For most production workloads serving 7B+ parameter models at modest batch sizes, the answer is yes.