Speculative decoding vs standard autoregressive decoding

Speculative decoding vs autoregressive decoding is the central latency optimization question for anyone serving LLMs at scale. Standard autoregressive decoding generates one token per forward pass, while speculative decoding uses a smaller draft model to propose multiple tokens that a larger target model verifies in parallel. The speedup can be 2–3× on typical workloads, but the engineering cost is real: you need two models, a verification kernel, and careful handling of acceptance rates. This post breaks down the mechanics, the trade-offs, and the decision framework we use when configuring inference pipelines.

How autoregressive decoding works

Standard autoregressive decoding is the baseline. Given a prompt, the model runs a forward pass to produce logits for the next token, samples or argmaxes one token, appends it to the sequence, and repeats. Each token requires a full forward pass through the entire model.

def autoregressive_generate(model, input_ids, max_new_tokens, temperature=1.0):
    for _ in range(max_new_tokens):
        logits = model(input_ids)[:, -1, :]  # (batch, vocab)
        probs = torch.softmax(logits / temperature, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        input_ids = torch.cat([input_ids, next_token], dim=1)
    return input_ids

The latency per token is dominated by memory bandwidth: loading model weights from VRAM into compute units. For a 70B parameter model at FP16, that’s ~140 GB of weight reads per token. On an H100 (3 TB/s bandwidth), the theoretical floor is ~47 ms/token — before kernel overhead, attention computation, and sampling. Real-world throughput lands around 20–30 tokens/second per GPU for 70B models.

The advantage is simplicity. One model, one code path, deterministic output (given a seed), and no quality degradation. Every optimization — quantization, flash attention, continuous batching, paged attention — applies directly.

How speculative decoding works

Speculative decoding adds a draft model that predicts k tokens ahead, then asks the target model to verify them in a single forward pass. The target model runs once on the extended sequence and checks whether its own predictions match the draft’s proposals. Matching tokens are accepted; the first mismatch triggers rejection, and generation resumes autoregressively from that point.

def speculative_generate(draft, target, input_ids, max_new_tokens, gamma=4):
    # gamma = number of draft tokens per iteration
    while len(input_ids[0]) < max_new_tokens:
        # Draft phase: generate gamma tokens autoregressively with small model
        draft_tokens = []
        draft_logits = []
        curr_ids = input_ids
        for _ in range(gamma):
            logits = draft(curr_ids)[:, -1, :]
            draft_logits.append(logits)
            token = torch.argmax(logits, dim=-1, keepdim=True)  # greedy draft
            draft_tokens.append(token)
            curr_ids = torch.cat([curr_ids, token], dim=1)
        
        # Verification phase: single forward pass on target
        target_logits = target(curr_ids)[:, -(gamma+1):-1, :]  # logits for draft positions
        
        # Acceptance: compare target vs draft distributions
        accepted = 0
        for i in range(gamma):
            draft_prob = torch.softmax(draft_logits[i], dim=-1)
            target_prob = torch.softmax(target_logits[:, i, :], dim=-1)
            # Acceptance probability: min(1, target_prob / draft_prob) for sampled token
            # Simplified: accept if argmax matches (greedy) or probabilistic acceptance
            if torch.argmax(target_prob, dim=-1) == draft_tokens[i]:
                accepted += 1
            else:
                break
        
        # Keep accepted tokens, resample from target at rejection point
        input_ids = curr_ids[:, :input_ids.shape[1] + accepted]
        if accepted < gamma:
            # Resample the rejected position from target distribution
            resample_token = torch.multinomial(target_prob, num_samples=1)
            input_ids = torch.cat([input_ids, resample_token], dim=1)
    
    return input_ids

The key insight: verification is a single forward pass regardless of gamma. If the draft model is 10–20× smaller (e.g., 7B drafting for 70B), its forward pass is negligible. The target model does the same compute as one autoregressive step but produces accepted + 1 tokens. Expected speedup is roughly 1 + gamma * acceptance_rate.

Acceptance rate depends on draft quality. A well-matched draft (same architecture, same tokenizer, trained on similar data) achieves 70–90% acceptance on typical text. Mismatched drafts — different tokenizer, different training distribution — can drop below 30%, killing the speedup.

Comparison table

Latency and throughput

The latency win comes from amortizing the target model’s memory bandwidth cost over multiple tokens. On an H100 with a 70B target and 7B draft (gamma=4, 80% acceptance):

• Autoregressive: ~25 tokens/sec → 40 ms/token
• Speculative: ~65 tokens/sec → 15 ms/token effective

But this assumes the draft model fits in VRAM alongside the target. If you’re already memory-bound (e.g., 70B at FP16 needs ~140 GB; 8×H100 gives 640 GB, leaving room), the draft is free. If you’re running 70B on 2×H100 with 4-bit quantization (~40 GB), adding even a 7B draft (14 GB at FP16, ~4 GB at 4-bit) may push you to offloading or a third GPU — negating the gain.

Throughput under continuous batching is trickier. Autoregressive decoding fits neatly into iterative batching: each iteration, every request in the batch produces one token. Speculative decoding produces a variable number of accepted tokens per request per iteration. You either pad to the maximum (wasting compute) or implement a more complex scheduler that re-inserts requests after variable progress. Most open-source implementations (vLLM’s speculative decoding, TGI’s draft model support) handle this with a “speculative batch” abstraction, but it adds scheduler complexity.

First-token latency (TTFT) is slightly worse with speculative decoding because the draft model must run its first forward pass before the target can verify. For short generations (< 50 tokens), the overhead can exceed the savings. Speculative decoding shines on long generations: code completion, document summarization, multi-turn chat.

Memory and compute trade-offs

The draft model must share the target’s tokenizer. This is non-negotiable — mismatched vocabularies make verification impossible without expensive mapping layers that destroy latency. In practice, this means:

• Same tokenizer (e.g., both Llama-2 tokenizer, both Mistral tokenizer)
• Compatible architecture (both decoder-only transformers)
• Similar training distribution for high acceptance

You can use a quantized draft (4-bit or even 3-bit) with minimal acceptance loss, but the target should stay at higher precision (FP16, BF16, or 8-bit). Quantizing the target aggressively changes its distribution, which reduces acceptance rate and can introduce subtle quality drift.

VRAM budgeting example for 70B target + 7B draft on H100 (80 GB):

At 4-bit, both fit on one H100 with headroom. At FP16, you need 3×H100. The draft’s memory cost is small relative to the target, but it’s not zero — and it scales with context length and batch size via its own KV cache.

Implementation complexity

Autoregressive decoding is a for-loop. Speculative decoding requires:

1. Draft model loading and warm-up — separate model instance, potentially different quantization
2. Verification kernel — fused kernel that computes target logits for draft positions and performs acceptance sampling in one pass (avoids materializing full logits for all vocab)
3. Acceptance logic — probabilistic acceptance (target_prob / draft_prob) for distributional correctness, or greedy matching for speed
4. Rejection handling — resample from target at rejection point, manage KV cache rollback for rejected tokens
5. Fallback path — when acceptance rate collapses (e.g., out-of-distribution prompts), gracefully degrade to autoregressive

The verification kernel is the performance-critical piece. A naive implementation runs the target model on the full extended sequence, then slices logits. A fused kernel computes only the needed logit positions and applies the acceptance criterion in-register. vLLM and TGI both ship optimized kernels for this; rolling your own is error-prone.

Probabilistic acceptance preserves the exact target distribution. Greedy acceptance (accept if argmax matches) is faster but introduces a small bias toward the draft model’s preferences. For most applications the bias is negligible, but if you need distributional fidelity — e.g., sampling for data generation, controlled generation with logits processors — you need the full acceptance ratio.

Quality and correctness guarantees

Speculative decoding with probabilistic acceptance is exactly equivalent to autoregressive decoding from the target model. The math: each token is accepted with probability min(1, p_target / p_draft), and rejected tokens are resampled from p_target. This is a standard acceptance-rejection sampling proof. The output distribution is identical to the target model alone.

Greedy acceptance breaks this guarantee. The bias is small when acceptance rate is high, but compounds over long generations. If you’re building a system where output distribution matters (e.g., synthetic data generation, evaluation harnesses), use probabilistic acceptance or stick to autoregressive.

There’s a subtlety with temperature and top-p/top-k sampling. The draft model should use the same sampling parameters as the target, or at minimum a temperature ≤ target temperature. If the draft is sharper (lower temperature) than the target, acceptance rate drops. If the draft is flatter, it proposes tokens the target assigns near-zero probability, also dropping acceptance. Match sampling configs.

When to choose each

Choose autoregressive decoding when:

• Short generations dominate — chat with < 100 token responses, classification, extraction. The draft warm-up and verification overhead exceed savings.
• Memory is tight — you’re already at the edge of VRAM with the target model alone. Adding a draft forces offloading or a smaller batch size, hurting throughput more than speculation helps.
• Heterogeneous model zoo — you serve 20+ models with different tokenizers/architectures. Maintaining matched draft models for each is operational burden with diminishing returns.
• Distributional fidelity is mandatory — synthetic data pipelines, benchmarking, any case where you must prove output matches the target exactly.
• Simplicity wins — small team, limited inference engineering bandwidth. The debugging surface of speculative decoding (acceptance rate drops, KV cache sync bugs, draft-target divergence) is real.

Choose speculative decoding when:

• Long generations are common — code completion, document generation, RAG with long answers, multi-turn reasoning. The amortized speedup compounds.
• You control the model pair — you can train or select a draft that matches the target’s tokenizer and distribution (e.g., Llama-3-70B target + Llama-3-8B draft, or a distilled draft).
• Batch size is moderate to high — continuous batching absorbs the variable token yield per request. At very low batch sizes (1–2), the scheduler overhead dominates.
• Latency budget is tight — you need sub-20ms/token on 70B-class models and have the VRAM headroom.
• You’re using a framework that ships it — vLLM, TGI, TensorRT-LLM have production-ready speculative decoding. The integration cost is a config flag, not a research project.

Hybrid approach: Run autoregressive by default, enable speculative decoding only for requests with max_tokens > 256 or explicit speculative=true header. This captures the wins on long generations without penalizing short requests. Most inference gateways support per-request routing directives for this.

The decision ultimately comes down to: do you have the VRAM headroom for a draft model, the engineering bandwidth to operate it, and a workload with enough long generations to amortize the cost? If yes, speculative decoding is a 2× latency win for ~15% more VRAM. If no, autoregressive with continuous batching, quantization, and flash attention gets you 80% of the way there with half the operational complexity.