Speculative decoding output quality is the central concern for any team adopting this acceleration technique. The promise is straightforward: a small draft model proposes tokens, a large target model verifies them, and you get 2-3x speedup with identical outputs. But “identical” carries weight. This post examines why the verification step preserves distribution fidelity, where the guarantees hold, and what actually breaks them in production.
The core thesis: verification is exact, not approximate
Speculative decoding does not approximate the target model’s distribution. It samples from it exactly. The draft model — whether a smaller transformer, a quantized version, or a specialized architecture like Medusa — proposes a sequence of tokens. The target model then evaluates each proposed token in parallel, computing the true conditional probabilities. Acceptance follows a precise probabilistic rule: accept token i if a uniform random draw falls below the ratio of target probability to draft probability (capped at 1). Reject, and you resample from the corrected distribution.
This is rejection sampling, not distillation. The output distribution matches the target model by construction, assuming correct implementation and floating-point determinism. The draft model only affects how many tokens you verify per forward pass, not which tokens survive.
# Simplified acceptance logic per token position
def accept_token(target_logits, draft_logits, temperature=1.0):
target_probs = softmax(target_logits / temperature)
draft_probs = softmax(draft_logits / temperature)
ratio = target_probs / draft_probs
accept_prob = torch.min(ratio, torch.ones_like(ratio))
return torch.rand_like(accept_prob) < accept_probIf the draft model assigns zero probability to a token the target model favors, the ratio becomes infinite and acceptance is guaranteed. If the draft model is overconfident on a token the target model dislikes, the ratio drops and rejection becomes likely. Either way, the marginal distribution at each position matches the target.
Why the draft model’s quality doesn’t degrade outputs
A common misconception: “If the draft model is bad, the outputs get worse.” This confuses throughput with quality. A poor draft model proposes tokens the target model rejects. You waste compute on verification and fall back to single-token generation for those positions. Latency increases, but the accepted tokens still follow the target distribution exactly.
Consider an extreme case: a draft model that outputs uniform random tokens. The target model rejects nearly all of them. You pay for one large-model forward pass per token — same as vanilla generation — plus the overhead of the draft forward pass. Quality is unchanged; you just lost the speedup.
The draft model’s role is proposal efficiency, not distribution shaping. This is why you can use aggressively quantized draft models (4-bit, even 2-bit) or tiny models (68M parameters drafting for 7B) without quality loss. The verification step corrects every deviation.
Where the guarantees hold — and where they bend
The theoretical guarantee assumes:
- Exact arithmetic — target and draft logits computed in the same precision
- Identical tokenizers — no vocabulary mismatch
- Correct temperature handling — both models apply temperature before softmax
- No early stopping divergence — EOS handling matches
In practice, three factors introduce subtle deviations:
Floating-point non-determinism
GPU kernels are non-deterministic across runs, devices, and batch sizes. The target model’s logits for a given prefix may differ by 1-2 ULP depending on whether you verify 1 token or 5 in parallel. This changes the acceptance boundary microscopically. Over thousands of tokens, the output distribution diverges from a pure sequential baseline — but also diverges from run to run of the baseline itself. This is not a speculative decoding artifact; it’s a property of floating-point on parallel hardware.
# Typical logit variance across runs (fp16, A100)
# Max abs diff: ~0.003
# KL divergence from sequential baseline: ~1e-6Temperature and top-p interaction
If the draft model applies top-p filtering before proposing, it zeroes out tokens the target model might have assigned non-zero probability. The ratio becomes 0/0 (undefined) or non-zero/0 (infinite). Implementations handle this differently: some clamp, some skip filtering in the draft, some resample. Each choice creates a measurable — though usually negligible — distributional shift.
Early acceptance heuristics
Some implementations (notably certain vLLM and TensorRT-LLM configurations) use “early accept” thresholds: if the target probability exceeds a high confidence bound, skip the exact ratio computation. This does approximate. The shift is typically <0.1% token-level KL but exists. Know your inference engine’s defaults.
Concrete example: Medusa heads vs. vanilla drafting
Medusa attaches multiple decoding heads to a frozen backbone, each predicting a future token position. The heads train to match the target model’s conditional distributions at their respective offsets. During inference, you get a tree of proposals instead of a linear chain.
{
"draft_structure": "tree",
"branching_factor": 4,
"depth": 3,
"total_proposals": 13,
"verification": "parallel_batched"
}Quality impact: Medusa heads are trained with a distillation loss that minimizes KL to the target model’s next-token distribution at each position. They are not trained on the joint distribution of the tree. The verification step still corrects each token individually using the true target conditionals. Output quality matches the target model. The only cost is training the heads — typically 1-2% of pretraining compute.
Contrast with “lookahead decoding” where a single small model rolls out linearly. Medusa’s tree structure yields higher acceptance rates (60-80% vs 30-50%) because multiple branches cover more probability mass. But the quality of accepted tokens is identical in both cases — verification is the equalizer.
The real quality risks: system-level, not algorithmic
If you ship speculative decoding and see quality regressions, check these first:
1. Tokenizer mismatch. The draft and target must share the exact same tokenizer instance — same vocab, same special tokens, same pre-tokenization rules. A draft model loaded from a different checkpoint with a “compatible” tokenizer will propose token IDs that map to different strings. Verification passes (the IDs match), but the decoded text diverges.
2. Prompt template drift. If your draft model expects a different chat template (e.g., no system prompt, different role markers), its proposals condition on a different effective prefix. The target model verifies against its prefix. The acceptance logic still works mathematically, but the draft is proposing for a different conversation history. Acceptance rates plummet; fallback to single-token generation hides the symptom.
3. KV cache inconsistency. Speculative decoding requires the target model’s KV cache to reflect only accepted tokens. If you accidentally commit rejected tokens to the cache (a known bug in early vLLM PRs), subsequent verification conditions on garbage. Outputs degrade catastrophically after the first rejection.
4. Batch-level interference. In continuous batching, a sequence that rejects early holds the batch slot while others continue. Some engines pad the rejected sequence’s KV cache with dummy tokens to maintain alignment. If those dummies leak into attention, quality drops. Verify your engine’s handling.
Measuring quality in your pipeline
Don’t trust benchmarks. Measure your specific workload.
def evaluate_speculative_quality(
target_model,
draft_model,
prompts,
num_samples=100,
max_tokens=512
):
results = {"sequential": [], "speculative": []}
for prompt in prompts:
# Baseline: pure target model
seq_out = target_model.generate(prompt, max_tokens, do_sample=True)
results["sequential"].append(seq_out)
# Speculative: same seed, same temperature
spec_out = speculative_generate(
target_model, draft_model, prompt, max_tokens
)
results["speculative"].append(spec_out)
# Compare distributions, not individual samples
# Use n-gram KL, semantic similarity, task-specific metrics
return compute_metrics(results)Key metrics:
- Token-level KL divergence between sequential and speculative outputs (target: <0.001)
- Task accuracy on your eval suite (MMLU, HumanEval, your internal benchmarks)
- Acceptance rate per position (diagnostic, not quality)
- Latency distribution (the actual payoff)
Run with fixed seeds across 50+ prompts. If KL is negligible and task metrics are flat, the implementation is sound.
When speculative decoding does change behavior
There is one legitimate case where outputs differ: deterministic decoding (temperature=0, greedy). Speculative decoding with greedy target and draft is not equivalent to pure greedy decoding. The draft model’s argmax may differ from the target’s argmax. Verification rejects the draft’s choice and the target’s argmax wins — but only after the draft proposed something else. The sequence of accepted tokens matches pure greedy if and only if the draft’s argmax equals the target’s argmax at every position. When they diverge, you get the target’s choice — but you paid for the draft’s mistake.
In practice, this means: do not use speculative decoding for temperature=0 workloads if you require bitwise identical outputs to a sequential baseline. Use it for sampling (temperature > 0) where the stochasticity masks the difference, or accept that greedy speculative is a distinct (and slightly faster) decoding algorithm.
The takeaway
Speculative decoding preserves output quality because verification is exact rejection sampling against the target model’s true distribution. The draft model affects throughput, not the marginal distribution of accepted tokens. Quality regressions in production come from implementation bugs — tokenizer mismatch, KV cache corruption, prompt template drift — not from the algorithm itself.
If you’re evaluating this for your inference stack: implement the verification logic yourself once (50 lines of PyTorch) to internalize the guarantees. Then audit your inference engine for the system-level pitfalls above. The speedup is real. The quality cost is zero — if you build it right.
For teams running heterogeneous model fleets, the routing layer matters. A gateway that can direct speculative workloads to GPU pools with matched draft/target pairs — and fall back to non-speculative when the draft model is unavailable — avoids the configuration drift that causes silent quality loss. That’s the infrastructure problem worth solving.