How a draft model speeds up speculative decoding

A draft model in speculative decoding is a smaller, faster language model that generates candidate tokens which a larger target model then verifies in parallel. The target model accepts or rejects each candidate in a single forward pass, yielding the same distribution as pure target-model sampling but with fewer expensive forward passes. This technique trades a small amount of extra compute on cheap hardware for large latency reductions on the bottleneck model.

How speculative decoding works

Speculative decoding relies on a simple probabilistic guarantee: if the target model assigns probability p to a token and the draft model assigns probability q, accepting the draft token with probability min(1, p/q) preserves the target model’s exact distribution. The algorithm proceeds in rounds:

1. The draft model generates k tokens autoregressively (typically k = 4–8).
2. The target model scores all k + 1 tokens (the prompt plus draft tokens) in one forward pass.
3. For each position i, compute the acceptance probability αᵢ = min(1, p_target(tokenᵢ | context) / p_draft(tokenᵢ | context)).
4. Accept tokens sequentially until the first rejection, then sample from the corrected distribution at that position and restart.

The key insight is that the target model’s forward pass is batched across all k candidates, so you pay one expensive forward pass to validate k tokens. When the draft model agrees with the target model—which happens frequently for predictable text—you get k tokens for the price of one.

# Minimal speculative decoding loop (PyTorch-like pseudocode)
def speculative_decode(draft_model, target_model, prompt, max_new_tokens=128, k=4):
    tokens = prompt.tolist()
    while len(tokens) < max_new_tokens:
        # 1. Draft model generates k candidates
        draft_tokens = []
        draft_logits = []
        ctx = tokens[:]
        for _ in range(k):
            logits = draft_model(torch.tensor([ctx]))[0, -1]
            probs = torch.softmax(logits, dim=-1)
            next_tok = torch.multinomial(probs, 1).item()
            draft_tokens.append(next_tok)
            draft_logits.append(logits)
            ctx.append(next_tok)

        # 2. Target model scores prompt + all draft tokens in one forward pass
        target_input = tokens + draft_tokens
        target_logits = target_model(torch.tensor([target_input]))[0]
        target_logits = target_logits[len(tokens)-1 : len(tokens)-1 + k]

        # 3. Accept/reject sequentially
        accepted = []
        for i in range(k):
            p_target = torch.softmax(target_logits[i], dim=-1)[draft_tokens[i]]
            p_draft = torch.softmax(draft_logits[i], dim=-1)[draft_tokens[i]]
            accept_prob = min(1.0, (p_target / p_draft).item())
            if random.random() < accept_prob:
                accepted.append(draft_tokens[i])
            else:
                # Rejection: sample from corrected distribution
                corrected = torch.clamp(target_logits[i] - draft_logits[i], min=0)
                corrected = torch.softmax(corrected, dim=-1)
                next_tok = torch.multinomial(corrected, 1).item()
                accepted.append(next_tok)
                tokens.extend(accepted)
                break
        else:
            # All k accepted
            tokens.extend(accepted)
    return tokens

This pseudocode omits KV-cache management and batched inference details, but the acceptance logic is exact. Production implementations fuse the target model’s forward pass across the entire k-token window and maintain separate KV caches for draft and target models.

Why the draft model matters

The draft model determines the acceptance rate, which directly controls speedup. Speedup ≈ 1 / (1 - acceptance_rate + 1/k). With k = 4 and 80% acceptance, you get ~2.5× throughput. With 50% acceptance, only ~1.6×. The draft model must be:

• Fast: Ideally 10–100× faster than the target model per token. This usually means 10–100× fewer parameters.
• Calibrated: Its probability distribution should correlate with the target model’s. A draft model that assigns high probability to tokens the target model rejects wastes compute.
• Cheap to run: Often deployed on the same GPU (using a separate stream) or a smaller accelerator.

Common draft model choices:

Distilled draft models are the sweet spot for most deployments. You train them by minimizing KL divergence between draft and target logits on your target domain data. A 7B draft for a 70B target typically reaches 70–85% acceptance on general chat, higher on code or structured output.

Concrete example: 70B target with 7B draft

Consider a 70B Llama-3 target model and a 7B draft model distilled on 10B tokens of your application data. On an H100 (80GB), the 70B model runs at ~2,500 tok/s (FP8, batch=1). The 7B draft runs at ~25,000 tok/s on the same GPU using a separate CUDA stream.

With k = 8 and 75% acceptance rate:

• Each round: 1 target forward pass (8 tokens scored) + 8 draft forward passes
• Wall time per round ≈ max(target_time, 8 × draft_time) ≈ target_time (draft is hidden)
• Effective throughput ≈ 2,500 × 8 × 0.75 = 15,000 tok/s

That’s 6× speedup. The draft model’s latency is completely hidden because it runs asynchronously. The only synchronization point is the acceptance check, which is negligible.

# Async draft generation with CUDA streams (conceptual)
draft_stream = torch.cuda.Stream()
target_stream = torch.cuda.Stream()

def async_speculative_step(draft_model, target_model, tokens, k=8):
    # Launch draft generation on draft stream
    with torch.cuda.stream(draft_stream):
        draft_tokens, draft_probs = draft_model.generate(tokens, k=k)
    
    # Target model scores on target stream (can overlap with draft)
    with torch.cuda.stream(target_stream):
        target_logits = target_model.score(tokens + draft_tokens)
    
    # Synchronize and accept/reject on host
    torch.cuda.synchronize()
    accepted = accept_reject(draft_tokens, draft_probs, target_logits)
    return accepted

In practice, you’d use a framework like vLLM, TensorRT-LLM, or SGLang that handles stream orchestration, KV cache sharing, and batched verification. The draft model runs continuously, keeping a buffer of candidates ready so the target model never waits.

Why it matters for production systems

Speculative decoding is one of the few techniques that reduces latency without degrading output quality. Quantization, pruning, and distillation all trade quality for speed. Speculative decoding preserves the target model’s exact distribution (up to floating-point non-determinism in the acceptance sampling).

For latency-sensitive applications—chat, coding assistants, real-time agents—this matters. A 70B model at 2,500 tok/s produces ~30 ms/token. At 15,000 tok/s, that’s ~5 ms/token. End-to-end latency for a 500-token response drops from 15 seconds to 2.5 seconds. Users notice.

The tradeoffs:

• Memory: You need VRAM for both models. A 70B FP8 + 7B FP8 fits on one H100 (80GB) with ~10GB headroom for KV cache.
• Complexity: Async execution, dual KV caches, and acceptance logic add failure modes. Debugging acceptance rate drops is harder than debugging a single model.
• Domain shift: A draft model distilled on general data degrades on specialized domains (legal, medical, proprietary code). Retrain or swap drafts per domain.

Common misconceptions

Misconception: The draft model must be a smaller version of the same architecture. False. The draft model can be any architecture that outputs a probability distribution over the same vocabulary. Recurrent drafts (RWKV, Mamba), n-gram models, or even a tiny transformer with different attention patterns all work. The acceptance criterion only requires p_draft(token) > 0 wherever p_target(token) > 0.

Misconception: Speculative decoding changes the model’s outputs. False, modulo floating-point non-determinism. The acceptance-rejection algorithm is mathematically exact: the marginal distribution of accepted tokens matches the target model’s distribution. Any difference comes from numerical precision in the softmax/logit computations, not the algorithm.

Misconception: Higher k is always better. Diminishing returns hit hard. The probability of accepting all k tokens is (acceptance_rate)^k. At 80% acceptance, k=4 gives 41% full-acceptance rounds; k=8 gives 17%. Larger k increases the target model’s batch size (good for utilization) but also increases the chance of early rejection, wasting the tail of the batch. k=4–8 is the practical sweet spot.

Misconception: You need a draft model for every target model. One draft model can serve multiple target models if they share a vocabulary. A 7B draft distilled on a mixture of 70B and 30B target outputs works reasonably for both. The acceptance rate drops slightly but avoids maintaining multiple draft checkpoints.

Misconception: Speculative decoding only helps throughput, not latency. It helps both. For batch=1, latency per token drops by the speedup factor. For batch>1, throughput increases because the target model processes more tokens per forward pass. The GPU utilization curve shifts right—you get higher throughput at the same latency, or lower latency at the same throughput.

When not to use it

Speculative decoding adds complexity. Skip it when:

• Your target model is already small (<13B) and fast enough.
• You’re memory-bound and can’t fit both models.
• Your workload is highly unpredictable (creative writing, open-ended reasoning) where acceptance rates drop below 40%.
• You need deterministic outputs (speculative decoding introduces sampling variance in the acceptance step, though you can fix the RNG seed).

For most production LLM serving stacks running 30B+ models, speculative decoding with a distilled draft model is the highest-ROI optimization available. It turns a memory-bound inference problem into a compute-bound one, and compute scales cheaper than memory.