Reported Grok 4 tokens per second numbers swing from optimistic provider dashboards to sluggish numbers in production logs. The metric only means something when you anchor it to a specific request shape, concurrency level, and context length—otherwise you are comparing apples to GPU clusters. This analysis breaks down what drives Grok 4 generation speed and how to measure it without fooling yourself.
Vendor Numbers Are a Function of Their Load Generator
Most published Grok 4 tokens per second figures come from a single warm request, a large batch, or a synthetic prompt that hits the model’s sweet spot. That tells you the hardware’s ceiling, not your user’s experience.
If the test harness opens 64 parallel streams and measures aggregate output, it reports cluster throughput. If it opens one stream with a 32-token prompt and 256-token completion, it reports best-case per-request latency. Neither maps to a production mix of 2k-token RAG contexts and sporadic traffic.
TTFT Is Not Throughput
Time to first token (TTFT) measures scheduling and prefill. Grok 4 tokens per second during decode is a different phase. A provider can have 300 ms TTFT but sustain 80 tokens/s once generation starts, or vice versa under queue pressure.
You must instrument both. A chatbot that streams feels responsive with 50 ms TTFT even at 20 tokens/s; a batch extractor cares only about total runtime.
Measuring Real Grok 4 Throughput
Spin up a loop that mirrors your traffic. Use the OpenAI-compatible endpoint (xAI exposes one) and stream with usage reporting.
import time, openai
client = openai.OpenAI(base_url="https://api.x.ai/v1", api_key="KEY")
prompt = "Summarize the tradeoffs of speculative decoding for MoE models."
start = time.perf_counter()
resp = client.chat.completions.create(
model="grok-4",
messages=[{"role": "user", "content": prompt}],
stream=True,
stream_options={"include_usage": True},
max_tokens=400,
)
tok_count = 0
for chunk in resp:
if chunk.usage:
tok_count = chunk.usage.completion_tokens
elif chunk.choices[0].delta.content:
pass # would stream to client
end = time.perf_counter()
gen_time = end - start # includes TTFT; subtract prefill if you have separate signal
print(f"{tok_count} tokens in {gen_time:.2f}s -> {tok_count/gen_time:.1f} Grok 4 tokens per second")This gives an end-to-end number. To isolate decode, capture the timestamp of the first content delta and compute from there to the final usage chunk.
start_decode = None
for chunk in resp:
if chunk.choices and chunk.choices[0].delta.content:
if start_decode is None:
start_decode = time.perf_counter()
if chunk.usage:
tok_count = chunk.usage.completion_tokens
end = time.perf_counter()
if start_decode:
decode_secs = end - start_decode
print(f"Decode rate: {tok_count/decode_secs:.1f} tokens/s")Run at Your Concurrency
A single stream hides batching wins. Launch N concurrent requests using asyncio or threads. Plot aggregate tokens/s against N. You will see a knee where added concurrency stops helping because the GPU is saturated or the provider throttles.
import asyncio, openai, time
async def one(client, sem):
async with sem:
t0 = time.perf_counter()
r = await client.chat.completions.create(
model="grok-4",
messages=[{"role":"user","content":"Write a SQL query for a ledger"}],
stream=True, stream_options={"include_usage":True}, max_tokens=200)
n=0
async for c in r:
if c.usage: n=c.usage.completion_tokens
return n, time.perf_counter()-t0
async def main(conc):
client = openai.AsyncOpenAI(base_url="https://api.x.ai/v1", api_key="KEY")
sem = asyncio.Semaphore(conc)
res = await asyncio.gather(*[one(client, sem) for _ in range(conc)])
total_tok = sum(r[0] for r in res)
total_sec = max(r[1] for r in res) # wall clock for batch
print(f"Concurrency {conc}: {total_tok/total_sec:.1f} agg tokens/s")
asyncio.run(main(8))Do not trust one run. Warm the endpoint with five discarded requests, then take the median of ten measurements. Variance across trials often exceeds 30% on shared infrastructure.
Factors That Shrink Your Numbers
Context Length and KV Cache
Grok 4, like other large transformers, allocates a KV cache proportional to prompt length. A 8k-token RAG context consumes memory that could otherwise hold batch entries. Throughput per request drops because the scheduler admits fewer sequences. If you reuse prefixes, cache hits restore speed—but only if the provider honors cache-control. When routing through a gateway such as n4n.ai, client cache directives are forwarded and per-token metering isolates the savings.
The usage object tells the story:
{
"usage": {
"prompt_tokens": 1843,
"completion_tokens": 412,
"total_tokens": 2255,
"cache_read_tokens": 1500
}
}A high cache_read_tokens value means the provider skipped prefill for most of your prompt. That lowers TTFT and indirectly improves observed Grok 4 tokens per second because the decode phase starts earlier.
MoE Architecture and Decode Bound
Recent frontier models, likely including Grok 4, use mixture-of-experts layers. Decode is memory-bandwidth bound, not compute bound. Activating a subset of experts per token keeps FLOPs modest but still moves large weight matrices across the bus. This means per-request tokens/s stays relatively flat as model size grows, while aggregate throughput scales with how many sequences fit in memory.
Quantization and Precision
Providers may serve Grok 4 at fp8 or lower precision to cut cost. That raises tokens/s but can shift output distribution. You cannot measure this from outside; you can only A/B output quality while watching throughput.
Provider Load and Fallback
During peak, a provider may deprioritize your requests. Your measured Grok 4 tokens per second at 2pm UTC will differ from 2am. Automatic fallback to a secondary region—if your client supports it—keeps p95 latency stable but mixes models. Know whether your gateway distinguishes model variants.
Latency vs Throughput Tradeoff
To maximize cluster throughput, providers pack batches. That raises TTFT for each request. If your app is interactive, you want bounded TTFT even at the cost of lower per-request tokens/s. Set max_tokens tightly and avoid gigantic system prompts.
If you run offline summarization, crank concurrency until the knee. There, Grok 4 tokens per second aggregate might double or triple versus serial calls, but individual jobs wait in queue.
Streaming vs Non-Streaming
Non-streaming lets the server batch more aggressively because it knows the full generation length upfront. Streaming forces incremental sends. The difference is usually small for decode rate but measurable for TTFT. Test both.
Reading the Knee of the Curve
Plot aggregate output rate versus concurrency. At low N, you see linear scaling: two streams give twice the tokens/s. Around the saturation point, the slope flattens. Beyond it, adding streams increases TTFT without raising total tokens/s. That knee is your capacity plan. For a single Grok 4 deployment, it might appear at dozens of concurrent sequences; for a shared tier, earlier.
Cost Per Token vs Speed
Higher throughput usually means lower cost per token because fixed overhead is amortized. But if you provision dedicated capacity to chase numbers, idle cycles burn money. Measure tokens/s per dollar, not just raw speed.
Honest Takeaway
Stop quoting headline Grok 4 tokens per second. Stand up a load test that replays your real prompt distribution and concurrency, measure decode rate separately from TTFT, and watch the curve as you increase parallel streams. The number that matters is the sustained generation rate at your p95 concurrency, not the vendor’s single-stream best. If you need cross-provider comparison, use a gateway that reports per-token usage and respects cache hints so the only variable is the model itself.