n4nAI
Open console →
Benchmarks & performanceComparisonServing Framework Benchmarks

vLLM vs TGI vs SGLang: throughput benchmark compared

A practical engineer's guide to the vllm vs tgi vs sglang benchmark tradeoffs across throughput, latency, ergonomics, and cost models for self-hosting LLMs.

n4n Team5 min read1,062 words

Audio narration

Coming soon — every post will get a voice note here.

The vllm vs tgi vs sglang benchmark question dominates any Slack thread about self-hosting LLMs. We’ve run all three behind production traffic and stripped away the marketing to show where each serving framework actually wins, and where it quietly loses.

Capabilities and Architecture

vLLM

vLLM came out of UC Berkeley with PagedAttention, a memory manager that slices the KV cache into non-contiguous pages like an OS virtual memory subsystem. That design lets it pack far more concurrent sequences onto one GPU than naive allocators. It ships an OpenAI-compatible API server, supports tensor parallelism (TP), pipeline parallelism (PP), speculative decoding, and quantized weights (AWQ, GPTQ, bitsandbytes). Model support tracks the Hugging Face transformers roster closely.

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --tensor-parallel-size 2 \
  --max-num-seqs 256

TGI

Text Generation Inference (TGI) is Hugging Face’s Rust+Python stack. A custom Rust router handles continuous batching, tokenization, and weight loading. It supports watermarking, guided generation via Outlines, and first-class Hub integration with private repos and signed URLs. Its generation loop is battle-tested in HF’s own hosted endpoints.

text-generation-launcher \
  --model-id meta-llama/Llama-3-8B-Instruct \
  --num-shard 2 \
  --max-batch-prefill-tokens 4096

SGLang

SGLang is two things: a frontend DSL for structured prompting and a runtime using RadixAttention. The runtime builds a radix tree of prompt prefixes and reuses KV cache nodes across requests that share text. For agentic loops or fixed system prompts, that reuse is the whole game. It also exposes an OpenAI-compatible server and supports TP.

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3-8B-Instruct \
  --port 30000 \
  --tp 2

Throughput and Latency Characteristics

Raw tokens/sec is workload-shaped. The vllm vs tgi vs sglang benchmark spreads narrow when you fix model, GPU, and sequence-length distribution, then widens again under real traffic.

vLLM’s PagedAttention keeps GPU compute saturated under chaotic request sizes because it never reserves contiguous KV space per sequence. Under decode-heavy chat on A100/H100, it typically edges out TGI’s continuous batcher. Prefill throughput is competitive but not magic—it still hits the memory-bandwidth wall on long contexts.

TGI has closed the gap with recent releases. Its Rust router imposes slightly higher scheduling overhead under extreme fan-out, but p99 latency stays tight because batch steps run on fixed cadences. It handles very large prefills cleanly when you tune --max-batch-prefill-tokens.

SGLang wins decisively when many requests share a long prefix. RadixAttention matches prefix nodes in a radix tree, so the second request past the same 2k-token system prompt pays near-zero prompt-processing cost. On synthetic agent traffic with repeated templates, throughput multiplies versus uncached baselines. On completely unique prompts, it performs like vLLM under the hood.

None of these frameworks break the bandwidth bound. For long-context decode, throughput is limited by KV cache reads, not FLOPS.

Cost Model and Resource Efficiency

Cost tracks memory footprint and utilization. vLLM’s paging reduces fragmentation, so you serve more sequences per GPU-hour. TGI’s allocator is improved but historically reserves more fixed memory. SGLang’s prefix cache can shrink required VRAM for prompt processing, letting you downsize the node for prefix-heavy apps.

All three support quantization to cut VRAM. Example for vLLM with AWQ:

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-3-8B-Instruct-AWQ \
  --quantization awq

On spot instances, vLLM and SGLang restart faster due to lighter launchers; TGI’s Rust router compiles once but boots slower. If you run multiple models, vLLM’s ability to swap inactive pages to CPU helps density. TGI expects one model per replica unless you front it with a custom router.

Ergonomics and Developer Experience

vLLM is Python-first. Its CLI --help is exhaustive, and the OpenAI client just works:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "ping"}]
)
print(resp.choices[0].message.content)

TGI expects HF conventions: point at a repo, it fetches weights, tokenizer, and generation config. Its Prometheus metrics are the cleanest of the three—ready-made saturation gauges save you a weekend of instrumentation.

SGLang’s differentiator is the programming model. You express constrained decoding and branching natively:

import sglang as sgl

@sgl.function
def quiz(s, subject):
    s += sgl.system("You are a tutor.")
    s += sgl.user(f"Quiz me on {subject}")
    s += sgl.gen("question", max_tokens=64)

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))
print(quiz.run(subject="vectors"))

If your app is a prompt graph rather than plain chat, that DSL removes a class of string-concatenation bugs.

Ecosystem and Integrations

TGI is the default backend for many HF Inference Endpoints and integrates with Hub signed URLs and model cards. vLLM is the de facto standard in research clusters, Ray Serve deployments, and LangChain self-host docs. SGLang is younger but already ships its own router and Kubernetes examples.

If you front these with a gateway, an OpenRouter-class layer such as n4n.ai can abstract the endpoints and apply fallback when a provider is degraded, but the engine’s local behavior still dictates your tail latency. All three honor OpenAI-compatible /v1/chat/completions, so swapping them behind a proxy is low-risk.

Hard Limits and Operational Gotchas

vLLM’s tensor-parallel scaling stalls past 8 GPUs for small models due to all-reduce overhead. TGI requires a Rust toolchain if you patch the router. SGLang’s radix cache is memory-bounded; under infinitely varying prefixes it degrades to vanilla vLLM behavior and eats RAM for the tree.

None support true server-side sessions; you resend context each call unless you build a cache layer. Provider cache-control hints (e.g., cache_control on system blocks) are forwarded by some gateways but not universally honored by the engines themselves. Debugging vLLM’s scheduler requires reading its logs at DEBUG level; TGI surfaces batch rejections via metrics; SGLang needs radix tree inspection for cache hits.

Head-to-Head Comparison

Dimension vLLM TGI SGLang
Core innovation PagedAttention Rust continuous batcher RadixAttention prefix cache
Throughput (mixed chat) High Medium-High High
Throughput (shared prefix) Medium Medium Very high
Ergonomics Pythonic CLI HF-native, metrics rich DSL for prompt graphs
Ecosystem Research standard HF Hub tight Growing
Quantization AWQ, GPTQ, BNB AWQ, GPTQ, BNB AWQ, GPTQ
Boot time Fast Slower (Rust compile) Fast
p99 latency Good Good Good (prefix wins)
Max TP scale 8+ (diminishing) 8+ 8+

Which to Choose

Self-hosted chat API with varied traffic: Run vLLM. Its paging handles messy production mixtures, and the OpenAI surface means client code doesn’t change when you swap models.

HF-centric shop with strict observability: TGI. If weights live on the Hub and you want Prometheus hooks without extra exporters, it’s the path of least resistance.

Agentic loops or fixed system prompts: SGLang. The radix cache turns repeated prefixes into near-free compute. For a RAG bot with a 4k-token template, that alone justifies the swap.

Multi-framework experimenter: Put a thin proxy in front and A/B them. The vllm vs tgi vs sglang benchmark you care about is your own traffic shape, not a public leaderboard.

You don’t want to operate GPUs: Use a gateway that fronts all three. The engine choice still matters upstream, but per-token metering and automatic fallback let you ignore the cold start.

Pick based on workload shape, not screenshot leaderboards. The framework that wins the benchmark you ran last week is the one closest to your own request distribution.

Tagsvllmtgisglangthroughputserving-frameworks

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceChoosing a serving framework: vLLM, TGI, or SGLang in 2026A practical guide to choosing vllm tgi sglang 2026: match serving frameworks to your workload, benchmark correctly, and avoid deployment pitfalls.Benchmarks & performanceHow PagedAttention changed LLM serving throughputPagedAttention rewired LLM serving to boost throughput via paged KV caches. This analysis covers how vLLM uses it, tradeoffs, and when to adopt it.Benchmarks & performanceHow speculative decoding compares across serving frameworksA head-to-head speculative decoding serving frameworks comparison of vLLM and TGI across capabilities, latency, cost, ergonomics, and limits for engineers.Benchmarks & performanceBenchmarking KV cache efficiency: vLLM vs SGLang vs TGIPractical comparison of kv cache efficiency vllm sglang tgi across capabilities, cost, latency, and ergonomics to help engineers pick a serving framework.Benchmarks & performanceBenchmarking vLLM's continuous batching against TGIA practical head-to-head look at vllm continuous batching vs tgi across scheduling, throughput, cost, ergonomics, and limits, with a verdict for self-hosting LLMs.Benchmarks & performanceTGI vs vLLM: which serves Llama 3 70B faster?A pragmatic head-to-head of TGI vs vLLM for serving Llama 3 70B: latency, throughput, cost, ergonomics, and which to deploy for your production workload.

All serving framework benchmarks posts →