How acceptance rate affects speculative decoding speedup

Speculative decoding speeds up LLM inference by having a small draft model propose tokens that a larger target model verifies in parallel. The speedup you actually get depends almost entirely on one metric: the speculative decoding acceptance rate — the fraction of drafted tokens the target model accepts without correction. This post derives the relationship, explains why acceptance rate varies across positions, temperatures, and model pairs, and shows what to optimize for in production.

The speedup formula

Speculative decoding works in rounds. Each round, the draft model generates k tokens autoregressively. The target model then scores all k + 1 tokens (the prefix plus k drafts) in a single forward pass. Tokens are accepted sequentially until the first rejection; the target model samples a replacement for that position, and the next round begins.

If the acceptance rate is α (0 ≤ α ≤ 1), the expected number of accepted tokens per round is:

E[accepted] = α + α² + α³ + ... + αᵏ = α(1 - αᵏ) / (1 - α)

For large k this converges to α / (1 - α). The expected tokens generated per target-model forward pass is therefore 1 + α / (1 - α) = 1 / (1 - α).

Let t_d be the draft model latency per token and t_t be the target model latency per forward pass (which scores k+1 tokens simultaneously). The wall-clock time per round is k·t_d + t_t. The effective throughput in tokens per second is:

throughput = E[accepted] / (k·t_d + t_t)

Speedup relative to vanilla target-model decoding (t_t per token) is:

speedup = throughput / (1/t_t) = t_t · E[accepted] / (k·t_d + t_t)

When draft latency is negligible (t_d ≈ 0), this simplifies to:

speedup ≈ E[accepted] = 1 / (1 - α)

This is the core result: speedup asymptotically approaches 1/(1-α). At α = 0.5 you get 2×. At α = 0.75 you get 4×. At α = 0.9 you get 10×. The curve is convex — small improvements in acceptance rate yield disproportionately larger speedups once you pass ~0.7.

def theoretical_speedup(alpha: float, k: int = 8, t_d_ratio: float = 0.05) -> float:
    """
    Speedup factor vs vanilla decoding.
    t_d_ratio = t_d / t_t (draft latency as fraction of target forward pass).
    """
    if alpha >= 1.0:
        return float('inf')
    expected = alpha * (1 - alpha**k) / (1 - alpha)
    return expected / (k * t_d_ratio + 1)

for a in [0.3, 0.5, 0.7, 0.8, 0.9, 0.95]:
    print(f"α={a:.2f} → speedup={theoretical_speedup(a):.2f}×")

α=0.30 → speedup=1.38×
α=0.50 → speedup=1.89×
α=0.70 → speedup=3.07×
α=0.80 → speedup=4.55×
α=0.90 → speedup=8.33×
α=0.95 → speedup=14.3×

The k parameter matters less than you’d think. With α = 0.8, increasing k from 4 to 16 only improves speedup from 4.2× to 4.8×. The bottleneck is acceptance rate, not horizon.

Why acceptance rate varies

Draft model quality

The draft model must approximate the target model’s distribution. A draft trained via knowledge distillation on target-model outputs achieves higher α than an off-the-shelf smaller model. Typical pairs:

Distillation closes the distribution gap. The draft learns to mimic the target’s preferences, not just predict next tokens generally.

Temperature and sampling

Higher temperature flattens the target distribution, making it harder for the draft to guess correctly. At temperature 0 (greedy), α is maximized because both models pick argmax. At temperature 1.0, α can drop 20–30 percentage points.

# Simplified: acceptance probability for a single token
# assuming draft and target logits are correlated Gaussians
import numpy as np

def acceptance_prob(temp: float, correlation: float = 0.8) -> float:
    # Toy model: correlated logits, acceptance when argmax matches
    # Real acceptance uses speculative sampling (token-level rejection)
    # This approximates the trend.
    return max(0.0, correlation - 0.3 * temp)

for t in [0.0, 0.3, 0.7, 1.0, 1.2]:
    print(f"temp={t:.1f} → α≈{acceptance_prob(t):.2f}")

temp=0.0 → α≈0.80
temp=0.3 → α≈0.71
temp=0.7 → α≈0.59
temp=1.0 → α≈0.50
temp=1.2 → α≈0.44

If your product requires high temperature for creativity, speculative decoding helps less. Consider lowering temperature or using a stronger draft.

Position in sequence

Acceptance rate is not constant across a generation. It tends to be:

• High at the start — prompts constrain the space; both models agree on obvious continuations.
• Lower in the middle — open-ended reasoning, multiple valid paths.
• Variable at the end — EOS token is easy; structured output (JSON, code) can be either easy (rigid syntax) or hard (semantic choices).

# Empirical pattern from a 7B→70B distilled pair on GSM8K
position_acceptance = [
    (0, 0.92), (10, 0.88), (50, 0.78), (100, 0.72),
    (200, 0.68), (500, 0.75), (1000, 0.85)
]

This means effective speedup varies during generation. A single average α obscures the tail latency on hard positions.

Prompt style and domain

Code and structured formats (JSON, YAML) often have higher α than free-form prose because syntax constraints reduce entropy. Domain-specific drafts (e.g., a draft fine-tuned on SQL) outperform general drafts on their domain.

Practical tradeoffs

Draft model size

Larger drafts → higher α but higher t_d. The optimal draft size balances these. For a 70B target, a 7B draft is a common sweet spot: t_d ≈ 0.03–0.05 t_t on modern kernels, while α reaches 0.8+ when distilled. A 1B draft runs faster but α drops to 0.6–0.7, often losing the speedup entirely.

def optimal_draft_size(target_size: int) -> int:
    """
    Rule of thumb: draft ≈ target / 10 for distilled pairs,
    target / 5 for base pairs.
    """
    return target_size // 10  # distilled

for t in [7_000_000_000, 13_000_000_000, 70_000_000_000]:
    print(f"Target {t/1e9:.0f}B → draft {optimal_draft_size(t)/1e6:.0f}M")

Target 7B → draft 700M
Target 13B → draft 1.3B
Target 70B → draft 7B

Horizon k

Increasing k gives diminishing returns. The marginal accepted tokens per additional draft slot is αᵏ⁺¹. At α = 0.8, the 5th token adds 0.33 expected accepts; the 10th adds 0.11. But larger k increases the target model’s batch size (scoring k+1 tokens), which can increase t_t if it exceeds hardware batch capacity.

Recommendation: Set k = 4–8. Profile t_t vs k on your hardware; stop when t_t starts scaling superlinearly.

Verification overhead

The target model must score k+1 tokens per round. With FlashAttention-2 and batching, this is nearly free up to k ≈ 8–16 on H100/A100. Beyond that, the KV cache for the draft prefix grows and memory bandwidth becomes the limiter. Some implementations cap k dynamically based on available KV cache slots.

When speculative decoding hurts

If α < 0.3, the overhead of draft generation + verification exceeds vanilla decoding. This happens with:

• Mismatched draft/target pairs (no distillation)
• High temperature (>1.0)
• Adversarial or out-of-distribution prompts
• Very small drafts (e.g., 100M → 70B)

Guardrail: Monitor α in production. Disable speculative decoding for requests where rolling α < 0.4 over the last 100 tokens.

class SpeculativeDecoder:
    def __init__(self, draft, target, k=8, min_acceptance=0.4):
        self.draft = draft
        self.target = target
        self.k = k
        self.min_acceptance = min_acceptance
        self.recent_accepts = []
    
    def maybe_fallback(self) -> bool:
        if len(self.recent_accepts) < 50:
            return False
        rate = sum(self.recent_accepts) / len(self.recent_accepts)
        return rate < self.min_acceptance

Measuring acceptance rate in production

Don’t rely on offline evals. Log per-request acceptance rate and correlate with latency, temperature, and prompt type.

{
  "request_id": "req_abc123",
  "model_pair": "llama-3.1-7b-draft → llama-3.1-70b-target",
  "temperature": 0.7,
  "tokens_generated": 512,
  "rounds": 89,
  "total_drafted": 712,
  "total_accepted": 503,
  "acceptance_rate": 0.706,
  "speedup_observed": 3.2,
  "speedup_theoretical": 3.4,
  "fallback_triggered": false
}

Track these aggregates:

• P50/P99 acceptance rate by temperature bucket
• Speedup vs theoretical — gap indicates verification overhead or kernel inefficiency
• Fallback rate — how often you disable speculation mid-request

The decisive takeaway

Acceptance rate is the single lever that matters. Everything else — horizon k, draft latency, verification optimization — is secondary. A distilled 7B draft at α = 0.85 beats an optimized 1B draft at α = 0.65 every time, even if the 1B draft is 5× faster per token.

Invest your engineering effort in:

1. Distillation quality — train the draft on target outputs with reverse KL or sequence-level objectives
2. Temperature-aware routing — use speculative decoding at low temperatures; fall back to vanilla at high temperatures
3. Online monitoring — disable speculation per-request when α collapses

The math is unforgiving: speedup = 1/(1-α). There is no workaround for low acceptance rate. If your draft model can’t match the target’s distribution, speculative decoding is negative ROI.