Speculative decoding is the most practical way to cut time-to-first-token and wall-clock latency for autoregressive LLM serving without changing model outputs. This speculative decoding serving frameworks comparison puts vLLM and TGI side by side on the dimensions that actually matter when you ship: what they support, what they cost in VRAM and dollars, and how they fail. Both implement the same research idea—a cheap draft proposes tokens, the target model verifies them in parallel—but the engineering tradeoffs diverge fast.
What speculative decoding buys you
The target model still produces the exact same distribution and sampled tokens as greedy or temperature sampling without speculation. The win is mechanical: instead of running one forward pass per token, you run one pass over a block of k proposed tokens and accept the prefix that matches the target’s own predictions. On code and structured generation where drafts are accurate, you get near-linear speedup on decode-bound requests. On noisy chat, speedup collapses toward 1x because most proposals are rejected.
The draft can be a smaller model, a lightweight n-gram matcher, or a trained head (Medusa). Each framework picks a different subset.
The contenders: vLLM and TGI
vLLM is a Python-centric serving stack built on PyTorch with a focused scheduler and paged KV cache. TGI is Hugging Face’s Rust+Python serving stack, optimized for production stability and tight HF model support.
Both ship Docker images and HTTP APIs. Both support continuous batching. The difference is in how you turn on speculation and what you sacrifice.
vLLM speculative config
vLLM exposes speculation through a speculative_config dict. N-gram speculation needs no extra model:
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3-8B",
speculative_config={
"method": "ngram",
"num_speculative_tokens": 5
}
)A draft-model setup points at a smaller checkpoint:
vllm serve meta-llama/Llama-3-8B \
--speculative-config '{"method":"draft_model","model":"meta-llama/Llama-3-1B","num_speculative_tokens":3}'TGI speculative launch
TGI takes a draft model path on the CLI. It loads both target and draft into the same process:
docker run --gpus all -p 8080:80 \
-e MODEL_ID=meta-llama/Llama-3-8B \
-e SPECULATIVE_DECODING=mata-llama/Llama-3-1B \
ghcr.io/huggingface/text-generation-inference:latestTGI does not offer n-gram or Medusa in the stable release line; it is draft-model only.
Capabilities matrix
| Dimension | vLLM | TGI |
|---|---|---|
| Draft methods | ngram, draft model, Medusa (partial) | draft model only |
| API surface | OpenAI-compatible server, Python lib | Rust HTTP server, HF client |
| Extra VRAM | +0 (ngram) or +draft size | +draft model size |
| Dynamic batching | Continuous, prefetch-aware | Continuous, static window |
| Config ergonomics | JSON/YAML or code | CLI flag / env var |
| Ecosystem | LangChain, Ray, standalone | HF Hub, Endpoints, Transformers |
| Hard limits | Draft vocab must match target | Draft tokenizer must match target |
Latency and throughput characteristics
Speculative decoding helps only when the bottleneck is per-token decode latency, not prompt prefill or network. Under a single low-concurrency stream, vLLM with n-gram on repetitive SQL or JSON can hit 2x tok/s. With a draft model on Llama-3-8B/1B, both frameworks land in similar ranges because the verification math is identical—the draft accepts 40–70% of tokens depending on task.
Under high batch sizes, the picture inverts. Drafting adds extra GPU kernels (draft forward + verify) that compete with the target’s own batched forward. vLLM’s scheduler can disable speculation per-request when batch occupancy crosses a threshold; TGI applies a global toggle. If you serve mixed traffic, vLLM’s finer control avoids throughput regression on saturated nodes.
Cost model and resource overhead
Self-hosted cost is GPU-hours. N-gram speculation in vLLM is essentially free: it scans recent tokens on CPU/device and adds negligible memory. Draft-model speculation in either framework loads a second model—for Llama-3-8B + 1B that’s roughly 2–3 GB VRAM overhead plus the draft’s own compute.
If you front these with an OpenRouter-class gateway such as n4n.ai, you still meter per token, so speculative decoding is pure latency win with no extra line-item cost. Behind the gateway, the provider eats the VRAM overhead; you just see lower latency at the same price.
For on-prem, calculate break-even: if speculation lifts decode throughput 1.8x on a $1.20/hr A10G, you serve 1.8x traffic for the same rent. If your traffic is prefill-heavy (long docs, short answers), the overhead never pays back.
Ergonomics: configuration and APIs
vLLM’s Python API is the path of least resistance for experiments. You can swap speculative_config in code and inspect acceptance rates via logging. The OpenAI server mode means existing clients (including curl) need zero changes—speculation is server-internal.
TGI’s ergonomics favor ops teams who already run HF stacks. One env var flips draft decoding on. But there is no in-process Python API for speculation; you must run the server and call HTTP. TGI’s response schema includes a speculation field in debug mode, which is useful for acceptance tracing but less programmable than vLLM’s callbacks.
Rolling back is simpler in vLLM: remove the config block, no restart of separate processes. TGI requires restart with different env.
Ecosystem and integration
vLLM plugs into Ray Serve, BentoML, and most LLM orchestration layers. Its OpenAI compatibility means it drops behind any proxy that speaks that protocol. Medusa support (though experimental) lets you load a single expanded checkpoint instead of a separate draft, which simplifies artifact management.
TGI is the default for Hugging Face Inference Endpoints and integrates with the Transformers tokenizer pipeline natively. If your CI already pulls models from HF and you use the HF router or TGI client, adding speculation is a one-line deploy change. It also has battle-tested gRPC and Rust concurrency, which some teams prefer over Python asyncio under heavy load.
Limits and failure modes
Speculative decoding fails silently if misconfigured. vLLM with a draft model whose vocabulary size differs from the target will raise a load-time assert; n-gram never fails but yields 1x speedup on non-repetitive text. TGI requires the draft tokenizer to be identical—using a different revision throws a mismatch error at boot.
Both frameworks disable speculation automatically when the proposal would exceed max sequence length. Neither supports speculative sampling with different temperature between draft and target; the draft must mimic target params. If you use penalizers (repetition, frequency) on the target, the draft ignores them, causing more rejections and wasted compute.
Medusa in vLLM is still flagged experimental; don’t bet production latency SLAs on it yet.
Which to choose
Pick vLLM if:
- You serve latency-sensitive, decode-bound traffic and want n-gram (zero-cost) speculation first.
- You need per-request control to turn speculation off under load.
- You’re building in Python or already use the OpenAI API shape.
Pick TGI if:
- Your stack is Hugging Face native and you deploy via Inference Endpoints or standard HF Docker.
- You have a well-matched small draft model and want stable Rust serving with minimal config surface.
- You don’t need n-gram or Medusa and can tolerate a global speculation toggle.
Pick neither (use a gateway) if:
- You want per-token billing and zero infra, and just want lower latency. A gateway that routes to providers running these frameworks gives you the speedup without the VRAM math—the speculative decoding serving frameworks comparison becomes someone else’s on-call problem.