n4nAI
Open console →
Benchmarks & performanceAnalysisGrok Performance Benchmarks

Grok 4 time-to-first-token benchmark

Engineering analysis of Grok 4 time to first token: how prefill, caching, and real load shape latency, with code to measure it and production guidance.

n4n Team4 min read830 words

Audio narration

Coming soon — every post will get a voice note here.

Grok 4 time to first token is the latency metric that decides whether your chat UI feels responsive or broken. Vendor numbers hide the dependence on prompt length, prefix caching, and concurrent load, so you need to benchmark in your own environment before trusting it for interactive workloads.

Why TTFT Beats Tokens/sec for Interactive Apps

Throughput in tokens per second matters for batch jobs. For a user staring at a cursor, the gap between hitting send and seeing the first character is the only number they feel. Grok 4 is a large frontier model; its decode speed is respectable, but its prefill cost is what spikes latency on long system prompts or retrieved context.

If you ship a coding assistant that injects a 4K-token repo skeleton, the Grok 4 time to first token will be dominated by processing that prefix, not by generating the answer. Ignore TTFT and you will mis-size instances or wrongly blame the network.

What Drives Grok 4 Time to First Token

Prefill Math

Autoregressive models process the entire prompt in a forward pass before emitting token one. The cost scales with prompt tokens and model width. A rough estimate:

ttft ≈ queue_delay + (prompt_tokens / prefill_throughput) + small_fixed_overhead

Prefill throughput on modern accelerators for a model like Grok 4 is high but finite. Under a quiet dedicated endpoint, a 1K-token prompt may clear prefill in a few hundred milliseconds. Push that same prompt to 8K tokens and concurrency up, and TTFT stretches into seconds. The relationship is linear until you hit batching limits.

Prefix Caching Is the Lever

xAI supports prompt prefix caching. If your system prompt or RAG context repeats across requests, the provider stores the KV cache for that prefix. A cache hit skips most of the prefill compute. In practice, this converts Grok 4 time to first token from a function of full prompt length to a function of the uncached suffix.

Set cache control markers on stable prefixes. In an OpenAI-compatible call:

{
  "model": "grok-4",
  "messages": [
    {"role": "system", "content": "You are a terse SQL expert.", "cache_control": {"type": "ephemeral"}},
    {"role": "user", "content": "Select * from users where active=true"}
  ]
}

Not every gateway forwards these hints. An OpenAI-compatible endpoint that honors client routing directives and forwards provider cache-control hints preserves the optimization end to end.

Concurrency and Batching

Shared inference pools batch requests to raise GPU utilization. Your request may wait behind others for a batch slot. At low QPS, Grok 4 time to first token is prefill-bound. At high QPS, it becomes queue-bound. This is why a single curl test lies: it measures the empty highway, not the rush hour.

Measuring It Yourself

Don’t trust a dashboard you didn’t instrument. Wrap a streaming call and record the clock at first delta. The same pattern works against xAI directly or any compatible proxy.

from openai import OpenAI
import time

client = OpenAI(base_url="https://api.x.ai/v1", api_key="YOUR_KEY")

start = time.perf_counter()
stream = client.chat.completions.create(
    model="grok-4",
    messages=[{"role": "user", "content": "Summarize: " + "lorem ipsum " * 500}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft = time.perf_counter() - start
        print(f"Grok 4 time to first token: {ttft*1000:.0f}ms")
        break

Run this across prompt sizes (512, 2K, 8K tokens) and at staggered concurrency (1, 5, 20 workers). Plot percentiles, not averages. P99 TTFT under your real traffic shape is the number that predicts churn.

When routing through an OpenAI-compatible gateway like n4n.ai, the client code is identical and you get automatic fallback to another provider if xAI is rate-limited, but the measurement methodology stays the same.

Gateway and Routing Tradeoffs

A gateway that aggregates 240+ models behind one endpoint simplifies experimentation. You can flip model to grok-4 or a smaller sibling without rewriting HTTP layers. The catch: each hop adds negligible but nonzero latency. If you measure TTFT at the client, that includes TLS, gateway queue, and provider round-trip. Separate the components by timing from a region-collocated worker.

Per-token usage metering is useful here. Correlate billed tokens with TTFT samples to see if large cached prefixes actually reduced prefill cost. If your cache hit rate is low, Grok 4 time to first token will stay high and you’re paying premium rates for latency you could fix with prompt engineering.

When Grok 4 Is the Right Call Despite Latency

Grok 4 earns its keep on tasks needing deep reasoning or broad knowledge where a smaller model fails. If the user expects a thoughtful answer and tolerates a “thinking” spinner, a 1.5s TTFT is fine. For autocomplete or inline suggestions, it is not.

Concrete tradeoff: a 2K-token RAG answer from Grok 4 might first token at 800ms cached, while a 8B-class model returns in 200ms but cites hallucinations. Measure both on your eval set. Don’t default to the biggest model because the benchmark looks good in a lab.

Honest Limitations of Any Benchmark

We did not publish absolute numbers here because they shift with region, instance type, and provider load. A number from a quiet us-east morning is not your 6pm eu-west. The defensible claim: Grok 4 time to first token is highly sensitive to prompt length and cache state, and degrades nonlinearly under concurrency. Treat any single figure as a lower bound.

Takeaway

Benchmark Grok 4 time to first token with your own prompts, at your own concurrency, with caching enabled on stable prefixes. Stream, measure P99, and route through a layer that respects cache hints. If your interactive surface can’t absorb the prefill cost, shrink the prompt or drop to a smaller model—don’t pretend the latency isn’t there.

Tagsgrok-4time-to-first-tokenlatency

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceGrok 4 Heavy vs Grok 4: speed and cost comparedA head-to-head engineering comparison of Grok 4 Heavy vs Grok 4 speed, cost, latency, and limits to help you pick the right xAI model for production workloads.Benchmarks & performanceGrok 4 inference speed across available providersAnalysis of Grok 4 inference speed across providers: latency, gateway overhead, and routing tradeoffs for engineers building on xAI's model via APIs.Benchmarks & performanceGrok 4 performance benchmark: cost per million tokensA practitioner's analysis of Grok 4 cost per million tokens: how pricing structure, caching, and output ratio drive real LLM inference economics.Benchmarks & performanceGrok 4 performance benchmark for reasoning tasksA practitioner's analysis of Grok 4 performance benchmark reasoning: what eval scores miss, how to test it reproducibly, and where it fits in production.Benchmarks & performanceGrok 4 performance benchmark on long-context promptsA practical Grok 4 performance benchmark long context analysis: throughput, latency, and quality tradeoffs for engineers shipping LLM pipelines at scale.Benchmarks & performanceGrok 4 performance benchmark under high concurrencyAnalyze Grok 4 performance high concurrency: where throughput saturates, tail latency behavior, and how to load test and mitigate limits with request hedging.

All grok performance benchmarks posts →