Grok 4 inference speed across available providers

Evaluating Grok 4 inference speed providers turns up a surprising conclusion: nearly every option routes to the same xAI-owned backend. The differences that matter are network proximity, gateway overhead, and how aggressively your client retries or falls back under load. If you expected wildly different token rates between vendors, you will be disappointed—and that is the point.

The reality of Grok 4 hosting

xAI trains and serves Grok 4 on its own clusters. Unlike Llama or Mixtral, the weights are not publicly downloadable for self-hosting. Third parties that list “Grok 4” are API aggregators proxying xAI’s endpoint, not independent GPU operators running the model locally.

That fact collapses the usual provider comparison. You are not choosing between different GPU fleets with different kernels or quantization schemes. You are choosing between different request paths to one fleet. A “provider” in this context is a routing layer, not a compute layer.

This matters because most latency optimization guides assume heterogeneous hardware. For Grok 4, the hardware variable is fixed. The software and network variables are not.

What “provider” means for Grok 4 inference speed

When engineers talk about Grok 4 inference speed providers, they usually mean one of three things:

• First-party xAI API (api.x.ai): the origin server.
• Aggregator gateway: a single OpenAI-compatible endpoint that forwards to xAI with unified auth and billing.
• Regional proxy or cache front: a layer that may add prompt caching or request coalescing before hitting xAI.

All three ultimately hit xAI’s inference servers. The token throughput per request is bounded by xAI’s serving stack. What changes is the time before the first token, the variance under concurrency, and the behavior when xAI returns a 429 or 503.

A direct call from a us-east VM to xAI’s us-east endpoint is the baseline. Any other path adds something: a TLS hop, a logging middleware, a region shuffle. The question is whether that addition buys you enough resilience to be worth it.

Measuring latency where it matters

Raw benchmark numbers from a provider’s status page tell you little. You need to measure time-to-first-token (TTFT) and inter-token latency from your deployment region, against your real prompt shape.

Below is a minimal Python snippet using the OpenAI SDK against two different base URLs. It streams and records timestamps.

import time
import openai

def ttft(base_url, api_key, prompt):
    client = openai.OpenAI(base_url=base_url, api_key=api_key)
    start = time.perf_counter()
    first = None
    stream = client.chat.completions.create(
        model="grok-4",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            first = time.perf_counter()
            break
    return (first - start) * 1000  # ms

# Direct xAI
print(ttft("https://api.x.ai/v1", KEY_XAI, "Explain Raft consensus"))
# Via gateway
print(ttft("https://gateway.example/v1", KEY_GW, "Explain Raft consensus"))

Run this from the same host you will use in production. A 30 ms extra hop from a gateway is irrelevant if the gateway gives you automatic fallback during xAI’s partial outages.

Concurrent load test

Single calls hide queueing. Under concurrency, xAI’s scheduler may delay some sequences. Test with parallel requests:

import asyncio
import openai
import time

async def timed_call(client, prompt):
    start = time.perf_counter()
    stream = await client.chat.completions.create(
        model="grok-4",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.perf_counter() - start

async def load_test(base_url, key, concurrency, prompt):
    client = openai.AsyncOpenAI(base_url=base_url, api_key=key)
    tasks = [timed_call(client, prompt) for _ in range(concurrency)]
    return await asyncio.gather(*tasks)

# results = asyncio.run(load_test(..., 20, "Summarize this"))

If the p95 TTFT balloons at 20 concurrent calls on direct xAI but stays flat on a gateway, the gateway is likely spreading load across xAI regions or retrying with backoff. That is a speed win disguised as overhead.

Inter-token variance

TTFT is only half the story. Grok 4 generates at a rate dictated by xAI’s batching. Under load, the inter-token gap widens. Capture it:

def token_gaps(base_url, api_key, prompt, n=20):
    client = openai.OpenAI(base_url=base_url, api_key=api_key)
    stream = client.chat.completions.create(
        model="grok-4",
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    gaps = []
    last = None
    count = 0
    for chunk in stream:
        c = chunk.choices[0].delta.content
        if c:
            now = time.perf_counter()
            if last:
                gaps.append(now - last)
                count += 1
                if count >= n:
                    break
            last = now
    return gaps

If the gateway adds a buffering proxy, you may see larger gaps even with similar TTFT. Test both.

Gateway overhead vs. backend throughput

A gateway that simply reverse-proxies adds one TLS termination and a short internal hop. In practice that is 5–20 ms for TTFT in the same cloud region. Cross-region calls (e.g., EU client to US xAI) dominate the latency budget.

Where gateways can hurt is when they implement synchronous logging or token metering on the hot path. A well-built one forwards the stream untouched. For example, a gateway like n4n.ai honors client routing directives and forwards provider cache-control hints, keeping the proxy thin. The backend throughput is identical. If Grok 4 processes sequences at a given rate on xAI’s accelerators, no aggregator changes that. They can only change whether your request gets queued behind someone else’s flood.

Fallback and routing as speed levers

Because there is essentially one backend, “fallback” does not mean switching to a different GPU fleet. It means the gateway detects xAI’s 429 or 503 and retries against a secondary xAI region or delays with backoff. That can turn a hard failure into a 200 ms slower success.

If you call xAI direct and get a 429, you must implement backoff yourself. With a gateway, you can send a routing hint:

{
  "model": "grok-4",
  "messages": [{"role": "user", "content": "Summarize this RFC"}],
  "route": {
    "prefer": ["xai-us-east"],
    "allow_fallback": true,
    "max_retries": 2
  }
}

This mirrors the routing directives that OpenRouter-class gateways accept as extensions. The point: your speed under degradation is a function of retry strategy, not provider raw speed. Grok 4 inference speed providers that offer smart routing will outperform a bare endpoint during incidents even if their baseline is marginally higher.

Cache-control and prompt reuse

xAI supports prompt caching on long system prompts. Gateways that forward cache_control markers preserve that benefit. If your gateway strips them, you pay full prefill cost on every call, which directly inflates TTFT for large contexts.

client.chat.completions.create(
    model="grok-4",
    messages=[
        {"role": "system", "content": LONG_CONTEXT,
         "cache_control": {"type": "ephemeral"}},
        {"role": "user", "content": "Question?"}
    ],
)

A provider that drops this field forces Grok 4 to reprocess the system prompt each time. That is a real, measurable Grok 4 inference speed penalty attributable to the provider path. Verify your gateway passes the field untouched before trusting its “compatible” claim.

Tradeoffs: direct vs. aggregated

Direct xAI API

• Pros: shortest path, no intermediary, full control of retries, no metered markup.
• Cons: single region, no unified metering across models, you own the 429 handling and region failover.

Aggregator gateway

• Pros: one key, per-token usage metering, automatic fallback, routing hints, forward cache-control.
• Cons: slight baseline overhead, potential for misconfigured caching if the gateway is poorly built.

For a startup shipping a feature fast, the gateway wins on operational simplicity. For a latency-critical system already running in us-east with strict tail-latency SLAs, direct may shave the hop. But the direct path demands you write and maintain backoff, region selection, and cache logic.

Decisive takeaway

Stop shopping for a “faster Grok 4 inference speed provider” as if they run different hardware. Pick the path with the lowest consistent latency from your region, verify it forwards cache-control, and implement or buy retry/fallback. If you want zero ops, use a gateway that proxies xAI with thin overhead; if you want absolute minimal hop, call xAI direct and write your own backoff. The model speed is fixed—your architecture chooses the wait.