Tokens per second (tok/s) is the rate at which a model emits completion tokens during decoding. What does tokens per second mean for your application: it is the generated output token count divided by the wall-clock duration of the generation phase, and it directly governs how fluid a streamed response feels and how long a worker stays blocked on a request.
How token throughput is measured
A transformer inference pass splits into two phases. Prefill consumes the input prompt and builds the key-value cache. Decode then produces tokens one step at a time, each step conditioned on the previous. The decode step is memory-bandwidth bound on the GPU, not compute bound, which is why tok/s barely improves when you bump tensor parallelism past a certain point on small models.
The standard formula isolates decode:
tok/s = generated_tokens / (time_of_last_token - time_of_first_token)Prefill latency is excluded because it scales with prompt size and is often overlapped with request routing. If you include prefill, you get an average that hides the interactive feel.
Streaming vs batched measurement
With streaming APIs you can record timestamps per chunk. Without streaming, you only have a total latency that mixes prefill and decode. Never compare a non-streaming “tokens per second” number to a streaming one.
from openai import OpenAI
import time
client = OpenAI() # defaults to OpenAI, swap base_url for any compatible gateway
start = time.time()
first_ts = None
n_tokens = 0
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain TCP congestion avoidance."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if first_ts is None:
first_ts = time.time()
n_tokens += 1
end = time.time()
if first_ts:
gen_duration = end - first_ts
print(f"{n_tokens} tokens in {gen_duration:.2f}s")
print(f"throughput: {n_tokens / gen_duration:.1f} tok/s")This prints the decode rate your client actually observed. Run it against production traffic, not just a benchmark prompt.
Why tokens per second matters for your app
Perceived responsiveness
Users do not stare at a token counter. They feel time-to-first-token (TTFT) and then the steadiness of the stream. A chat UI rendering 40 tok/s feels snappy; one rendering 12 tok/s feels like it is stalling, even if the total answer arrives in acceptable time. For long generations—SQL drafts, code completions, agent trajectories—low tok/s multiplies into multi-second waits that break flow.
Worker occupancy and concurrency
If your service calls the LLM synchronously inside a request handler, the handler is pinned for the entire decode. At 20 tok/s, a 600-token response holds a thread for 30 seconds. At 100 tok/s, it is 6 seconds. That difference dictates how many concurrent requests a single backend worker can absorb before you need to scale out or move to async queues.
Cost is not reduced, but density is
Providers bill per output token regardless of speed. Higher tok/s does not cut your bill for a fixed task. It does let you pack more sequences onto a GPU via continuous batching, which is why self-hosted deployments care about throughput per dollar, not just per token.
A concrete example: streaming a summarization feature
Assume a feature that summarizes support tickets into a 400-token internal note.
- At 25 tok/s: generation takes 16.0s.
- At 80 tok/s: generation takes 5.0s.
- At 10 tok/s (degraded shared instance): generation takes 40.0s.
If your frontend has a 30-second timeout, the 10 tok/s backend silently fails half your requests. The 25 tok/s backend survives but triggers retry storms under load because users abandon. Only the 80 tok/s backend keeps p95 generation inside a comfortable modal window.
This is why load tests must record tok/s per request, not just error rates. A provider can be “up” while delivering tok/s that violates your UX contract.
# crude curl timing for a non-streaming call
curl -s -o /dev/null -w "total: %{time_total}s\n" \
-H "Authorization: Bearer $KEY" \
-d '{"model":"mistral-7b","messages":[{"role":"user","content":"Summarize: ..."}]}' \
https://api.example.com/v1/chat/completionsSubtract prefill estimate if you have it; otherwise treat the number as a lower bound on decode rate.
Common misconceptions about tok/s
“It is a fixed property of the model”
False. The same weights on an A100 with 80GB and on a shared T4 yield wildly different decode rates. Quantization, batch size, sequence length, and KV cache fragmentation all move the number. A 70B model on a single consumer GPU may crawl at 8 tok/s; the same model on a properly provisioned cluster may hit 60 tok/s.
“Higher tokens per second means lower total latency”
Only if TTFT is equal. A endpoint advertising 200 tok/s but taking 4 seconds to emit the first token will feel slower for a 30-token answer than a endpoint at 50 tok/s with 150ms TTFT. Always pair tok/s with TTFT when comparing providers.
“Input tokens count toward the rate”
They do not. Tok/s measures output generation. Input tokens are processed during prefill, which is typically measured in tokens per second of prompt processing—a different metric that rarely appears in marketing sheets.
“Published benchmarks match your workload”
Synthetic benchmarks use short prompts and empty KV caches. Production prompts carry long system instructions and few-shot examples. Cache hits skip prefill entirely. When you revisit what does tokens per second mean under cache hits, the decode rate is unchanged but the user-perceived latency drops because the waiting period before first token vanished.
Routing, fallback, and consistent measurement
When you front multiple vendors with a single OpenAI-compatible endpoint, the backend that serves a given request can change mid-traffic. A gateway that provides automatic fallback when a provider is rate-limited or degraded will silently reroute your call to a different GPU fleet. Your client-side measurement then reflects the throughput of whichever backend won. n4n.ai forwards provider cache-control hints and honors routing directives across 240+ models behind one endpoint, so if you cache prefill on one provider but fall back to another, your TTFT jumps even if the decode tok/s stays similar.
To keep numbers comparable:
- Tag each response with the
x-providerheader if your gateway exposes it. - Store TTFT and decode tok/s separately in your metrics pipeline.
- Alert on tok/s percentile drops, not just on 5xx errors.
What does tokens per second mean in that dashboard? It is the decode throughput of the winning backend for that specific request shape, nothing more universal than that.
Practical takeaways
- Measure tok/s from the client, with streaming, on real prompts.
- Track TTFT and decode tok/s as distinct SLOs.
- Load test at your actual concurrency; throughput collapses under batch contention.
- Treat provider-published tok/s as an upper bound under ideal conditions.
- If you use a gateway with fallback, attribute metrics per backend or you will average away outages.
Engineers who internalize this stop chasing headline numbers and start building timeout, retry, and queueing logic around the tok/s their users actually receive.