Llama 4 inference speed: Groq vs Cerebras vs Together

Evaluating Llama 4 inference speed Groq vs Cerebras vs Together is less about a single tokens-per-second number and more about how each system’s architecture maps to your request pattern. Groq’s LPUs, Cerebras’s wafer-scale engine, and Together’s optimized GPU fleets each make different tradeoffs that surface only under production load.

What actually drives Llama 4 inference speed

Raw benchmark figures for Llama 4 are not yet stable across providers, but the hardware determinants are clear. Groq ships tensor streaming processors (LPUs) that eliminate HBM bottlenecks for small-batch decode. Cerebras uses a single-wafer engine with massive on-chip memory bandwidth, favoring large batch and long-sequence throughput. Together runs quantized and speculative-decoded Llama variants on H100/H200 clusters, balancing elasticity with per-GPU economics.

The Llama 4 inference speed Groq vs Cerebras vs Together gap narrows when you consider that Llama 4 is expected to ship both dense and Mixture-of-Experts (MoE) variants. MoE shifts compute to expert routing; wafer-scale and LPU architectures handle static graphs differently than CUDA kernels under dynamic expert selection.

Groq: deterministic low latency

Groq’s compiler statically schedules the model graph. For Llama-class models, this yields predictable single-digit millisecond token intervals when batch size stays at one or two. The ceiling appears when you need many concurrent streams; the LPU fabric is not designed for high fan-out. You get fast TTFT but queueing kicks in past concurrency limits.

Cerebras: throughput monster

Cerebras exposes the model on a wafer-scale cluster. Published numbers for Llama 3.1 70B exceeded 1,800 tokens/s for a single sequence; Llama 4 will follow similar scaling if parameter count stays in the same regime. If your job is ingesting thousands of documents through one model copy, this is the fastest path. Interactive latency is good but not as razor-thin as Groq’s.

Together: flexible GPU fleet

Together gives you standard CUDA optimizations: continuous batching, tensor parallelism, and optional fine-tunes. Llama 4 inference speed on Together tracks what you’d get self-hosting on H100s, minus ops overhead. You trade peak single-stream speed for capacity and model variety.

Capabilities

All three serve Llama 4 (70B, 8B, and MoE variants as released). Groq and Cerebras focus on inference-only endpoints; Together additionally offers fine-tuning jobs and dedicated deployments.

• Groq: Llama 4 chat and base, fixed context window (typically 8k–128k depending on variant), no custom weights.
• Cerebras: Llama 4 inference with extended context options, beta function-calling on some sizes.
• Together: Full suite—instruction, base, fine-tuned, and LoRA serving; you can upload adapters and run custom checkpoints.

Context windows are provider-documented; do not assume parity. If your app needs 256k context, verify before committing.

Price and cost model

None of the three bills by hour; all use per-token metering. Groq positions itself as low-cost inference at speed. Cerebras prices for premium throughput, often higher per token for the same model size. Together sits in between with spot and reserved tiers.

If you front these with n4n.ai, you get per-token usage metering unified across all three and automatic fallback when a provider is rate-limited or degraded—useful when Llama 4 traffic spikes.

Token accounting

# Groq, Cerebras, Together all return usage in OpenAI format
resp = client.chat.completions.create(
    model="llama-4-70b",
    messages=[{"role": "user", "content": "summarize"}]
)
print(resp.usage.completion_tokens)  # billed units

Hidden costs: prompt caching. Together and Cerebras forward cache-control hints; Groq supports prefix caching on some models. A gateway that honors client routing directives and forwards provider cache-control hints preserves those savings across providers.

Latency and throughput

Latency splits into time-to-first-token (TTFT) and inter-token latency (ITL). Groq wins TTFT for single user. Cerebras wins sustained throughput for batched jobs. Together varies with cluster load but offers decent p50 ITL under 30ms for 70B on H100.

Measuring Llama 4 inference speed Groq vs Cerebras vs Together under batch load changes the ranking. At batch 1, Groq leads. At batch 64, Cerebras leads by an order of magnitude. Together scales linearly with allocated GPUs.

Concrete numbers from prior Llama generations: Groq Llama 3 70B showed ~300 tokens/s stream with <200ms TTFT. Cerebras Llama 3.1 70B hit >1,800 tokens/s single stream. Together Llama 3.1 70B hovered ~120 tokens/s on shared instances. Llama 4 will shift these but the ordering holds.

Ergonomics

All three expose OpenAI-compatible REST. That means one client code path, different base_url.

from openai import OpenAI

groq = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=GROQ_KEY)
cerebras = OpenAI(base_url="https://api.cerebras.ai/v1", api_key=CEREBRAS_KEY)
together = OpenAI(base_url="https://api.together.xyz/v1", api_key=TOGETHER_KEY)

for cli in (groq, cerebras, together):
    r = cli.chat.completions.create(
        model="llama-4-70b",
        messages=[{"role": "user", "content": "ping"}],
        max_tokens=8
    )
    print(r.choices[0].message.content)

Groq and Cerebras require no extra headers. Together supports vendor- extensions for fine-tuned model routing. All honor temperature and stream. Streaming responses use SSE identically; you can swap providers in a config file.

Ecosystem and limits

• Groq: Tight model catalog, frequent rate limits on free tier, enterprise tiers raise concurrency. No custom weight hosting.
• Cerebras: Fewer models overall, but deep integration for long context; rate limits per wafer allocation.
• Together: Largest model zoo, community adapters, but noisy-neighbor variance on shared endpoints.

Model refresh lag: Groq and Cerebras typically onboard Llama 4 within days of Meta release; Together often has it at launch plus extra quantized builds.

Head-to-head comparison

Which to choose

Real-time chat or agent loops

Pick Groq. The deterministic ITL keeps interactive agents snappy. If Groq rate-limits, route fallback to Cerebras.

Document ingestion or eval sweeps

Cerebras wins. When you need to push millions of Llama 4 tokens through a single pipeline, the wafer-scale throughput cuts wall-clock time.

Prototyping with custom weights

Together is the only one of the three that lets you fine-tune or attach LoRAs to Llama 4. Use it for experiments before committing to a speed tier.

Multi-provider production

Use a gateway that honors client routing directives and forwards cache-control hints. You keep one endpoint, get fallback, and meter per token across Groq, Cerebras, and Together without rewriting clients.

The Llama 4 inference speed Groq vs Cerebras vs Together decision is therefore not “who is fastest” but “which bottleneck is yours”: interactive latency, batch throughput, or weight flexibility. Choose accordingly.