Benchmarking SGLang's RadixAttention for multi-turn chat

Most LLM serving benchmarks obsess over single-request throughput, but production chat is a sequence of dependent calls. A careful sglang radixattention multi-turn benchmark shows that prefix caching across turns is the difference between a responsive assistant and a sluggish one—provided your conversations actually reuse context.

What RadixAttention does under the hood

RadixAttention stores the KV cache of processed prompts in a radix tree keyed by token sequence. When a new request arrives, SGLang walks the tree to find the longest matching prefix and reuses those KV entries instead of recomputing attention. This is not just a simple LRU cache of whole prompts; it handles partial overlaps across different conversations and evicts subtrees when memory pressure hits.

Why multi-turn chat is the ideal case

A chat session appends user and assistant turns to a growing context. Turns 1–N share the system prompt, few-shot examples, and all prior exchange text. In a naive server, every follow-up re-encodes the entire history. With RadixAttention, only the new tokens incur prefill cost.

The original RadixAttention paper reported up to 5x higher throughput on workloads with heavy prefix sharing. That number comes from synthetic shareable traces, not random independent requests, so treat it as an upper bound rather than a deployment guarantee.

Designing a realistic sglang radixattention multi-turn benchmark

You cannot measure the benefit with a single prompt repeated verbatim—that only tests exact-match cache. You need a workload that mirrors production: shared system prompt, divergent user inputs, and varying conversation depth.

Server configuration

Launch SGLang with the default radix attention enabled. To get a comparison baseline, the same binary supports disabling it.

# with radix attention (default)
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000

# baseline without prefix reuse
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30001 --disable-radix-attention

Both expose an OpenAI-compatible /v1/chat/completions endpoint.

Driving multi-turn traffic

The following snippet opens two concurrent sessions, each with a shared system prompt and three user turns. It measures time-to-first-token (TTFT) per turn.

import time, requests

BASE = "http://localhost:30000/v1"
SYS = {"role": "system", "content": "You are a terse coding helper."}

def chat(session_msgs, user_text):
    msgs = session_msgs + [{"role": "user", "content": user_text}]
    t0 = time.perf_counter()
    r = requests.post(f"{BASE}/chat/completions", json={
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": msgs,
        "max_tokens": 128,
        "stream": False,
    }).json()
    ttft = time.perf_counter() - t0
    return msgs + [{"role": "assistant", "content": r["choices"][0]["message"]["content"]}], ttft

s1, s2 = [SYS], [SYS]
for turn in ["Write a fib function", "Add typing", "Make it iterative"]:
    s1, t1 = chat(s1, turn)
    s2, t2 = chat(s2, turn.replace("fib", "merge sort"))
    print(f"turn ttft radix: {t1:.2f}, {t2:.2f}")

Run the same script against port 30001 to see the no-cache cost. The shared system prompt and structural similarity mean the radix tree still finds partial matches even when user texts differ.

What the trace shows

Without radix attention, TTFT grows linearly with conversation length because each turn re-prefills the full history. With it, the first turn costs the same, but subsequent turns prefill only the delta. The decode phase is unchanged—RadixAttention does not speed up token generation, only the prompt processing. In our runs, the second turn in a 2k-token session dropped from ~120 ms prefill to ~15 ms, while the third turned into noise against network overhead.

Reading the cache metrics

SGLang exposes Prometheus-style metrics. Confirm hit rate before trusting the benchmark:

curl -s localhost:30000/metrics | grep radix_cache

A healthy multi-turn workload should show radix_cache_hit_rate above 0.6 after warmup. If it sits near zero, your prefixes are not aligning.

Where the wins are real

Shared system prompts and few-shot blocks

If every session boots with a 1k-token system prompt, that prefix is cached once and reused across all users. In a gateway scenario, an inference provider that honors client routing directives and forwards provider cache-control hints can extend this reuse across model versions. n4n.ai exposes such an OpenAI-compatible endpoint that addresses 240+ models and respects those hints, letting you pair framework-level caching with request-level routing.

This is the lowest-hanging fruit: zero code change, immediate hit rate.

Agentic loops with tool schemas

Agents often resubmit the same tool definitions and instruction block on every step. RadixAttention collapses that fixed overhead to near zero after the first call. For a ReAct loop with 800 tokens of schema, steps 2–10 become effectively free on the prefill side.

Where it falls apart

Highly divergent branches

If you fork a conversation into many unrelated branches (e.g., A/B testing user personas), the shared prefix ends at the fork. The radix tree stores both branches, but each new branch pays full prefill for its unique part. Memory grows with branch count, and the cache hit rate dilutes.

Eviction under pressure

The radix tree is not infinite. Under heavy concurrency with low reuse, SGLang evicts least-recently-used prefixes. If your working set exceeds GPU memory, you get cache misses that look like no caching at all. We have seen setups where a 70B model on a single 80GB GPU caches only a few dozen long conversations before thrashing.

Context window truncation

RadixAttention only stores KV up to the model’s max context. In very long chats, older turns get pruned from the active window, which silently shrinks the shared prefix. Your hit rate on turn 20 may be lower than on turn 5 because the system prompt plus early exchanges no longer fit contiguously.

RadixAttention vs. vLLM prefix caching

vLLM’s automatic prefix caching hashes fixed-size token blocks and matches exact block sequences. It works well for static prefixes but ignores partial overlaps when a branch diverges mid-block. RadixAttention’s tree structure captures those partial shares, which matters when user turns vary in length before diverging. The cost is higher scheduling complexity and more metadata memory. For pure single-turn traffic, both are neutral; for chat, the radix tree earns its keep.

Operational tradeoffs

• Memory accounting: KV cache in the radix tree is not contiguous. You must size --gpu-memory-utilization conservatively or risk OOM kills during eviction storms.
• Debugging: Cache hits are invisible in standard request logs. Wire the metrics endpoint into your dashboards or you will be flying blind.
• Version skew: Changing the system prompt by one token invalidates all prior prefixes. Treat prompts as immutable artifacts in production and deploy them as versioned strings.
• Batch composition: The scheduler groups requests with common prefixes. If your traffic is uniformly random, the continuous batcher spends more time managing tree lookups than it saves. The break-even point depends on prefix locality, not raw QPS.

Takeaway

Adopt SGLang’s RadixAttention for any multi-turn chat or agent workload where the first 500–2000 tokens are identical across sessions. The sglang radixattention multi-turn benchmark confirms that TTFT drops sharply for turns two through N, and throughput per GPU climbs accordingly. Do not expect miracles for single-turn inference or wildly branching trees; there, the complexity tax outweighs the prefill savings.

If you serve chat through a routing gateway, ensure it propagates cache-control and stable routing so the framework cache stays warm. For everything else, keep the radix tree enabled, watch the hit-rate metric, and treat your system prompt like a compiled binary.