Tokens per second benchmark on Groq, Cerebras, and SambaNova

If you are optimizing for generation speed, the tokens per second Groq Cerebras SambaNova race matters more than raw FLOPS. These three vendors ship custom silicon purpose-built for transformer inference, and each exposes cloud endpoints that blow past typical GPU serving. This head-to-head compares them on the dimensions that actually affect production workloads.

Throughput: Tokens per Second in Practice

The headline metric is tokens per second Groq Cerebras SambaNova deliver on equivalent models. All three publish impressive numbers, but real-world throughput depends on model size, batch size, prompt length, and whether you are hitting a warm cache.

Groq’s LPU (Language Processing Unit) is a deterministic, single-tenant-per-chip design. For Llama-3-70B, community measurements consistently show output rates above 200 tokens/s, often approaching 300 tokens/s for small batches. Cerebras’ Wafer Scale Engine (WSE) uses massive on-wafer memory bandwidth; its published benchmarks for Llama-3.1-8B exceed 1,000 tokens/s per stream, and 70B-class models land in the same ballpark as Groq. SambaNova’s RDU (Reconfigurable Dataflow Unit) is more flexible for training and fine-tuning, but its inference throughput typically trails the other two by 20–40% on similar model classes.

Measurement is straightforward. Point an OpenAI-compatible client at the endpoint and time the stream:

import time, openai

def measure(base_url: str, api_key: str, model: str, prompt: str):
    client = openai.OpenAI(base_url=base_url, api_key=api_key)
    start = time.time()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
        max_tokens=512,
    )
    tokens = 0
    for chunk in stream:
        if chunk.choices[0].delta.content:
            tokens += 1
    elapsed = time.time() - start
    return tokens / elapsed

print(measure("https://api.groq.com/openai/v1", "GROQ_KEY", "llama3-70b-8192", "Explain TCP."))

Run the same function against Cerebras (https://api.cerebras.ai/v1) and SambaNova (https://api.sambanova.ai/v1) with their model IDs. You will see the tokens per second Groq Cerebras SambaNova gap directly.

What Skews the Numbers

Prompt caching, max tokens requested, and concurrency all distort benchmarks. A 1-token prompt with a 2K generation favors Cerebras and Groq; SambaNova’s strength appears when you reuse KV-cache across many short requests. Batching multiple sequences on one request is not uniformly supported—Groq expects you to open parallel streams, while Cerebras offers a dedicated batch endpoint.

Capabilities and Model Support

Groq focuses on inference-only. Supported models are a curated set: Llama 3 family, Mixtral 8x7B, Gemma, and a few others. Context windows top out at 8K–128K depending on model.

Cerebras mirrors that scope but adds larger context for some models (up to 32K or more) and supports speculative decoding for even higher token rates. It also offers a “batch” mode for offline jobs where you post a file and poll for results.

SambaNova differentiates with fine-tuning and training on the same fabric. You can upload a dataset and train a LoRA without leaving the platform. Inference supports Llama, Mistral, and proprietary Samba models, with context up to 64K. If your product needs a custom weight snapshot served at speed, only SambaNova covers the full loop.

Price and Cost Model

None of the three uses GPU-hour billing. They charge per output token, with input tokens sometimes free or heavily discounted.

Groq prices Llama-3-70B at roughly $0.59 per million output tokens (public list). Cerebras lists similar per-token rates but offers committed-use discounts for reserved capacity. SambaNova sits slightly higher per token for equivalent models, reflecting its broader feature set.

If you need to control spend precisely, meter at the client. An OpenAI-compatible gateway such as n4n.ai fronts 240+ models, applies per-token usage metering, and will automatically fallback when a provider is rate-limited or degraded—useful when you blend these vendors behind one endpoint.

Ergonomics and API Surface

All three expose OpenAI-compatible REST endpoints. That means one SDK, minimal branching:

def get_client(vendor: str, key: str):
    bases = {
        "groq": "https://api.groq.com/openai/v1",
        "cerebras": "https://api.cerebras.ai/v1",
        "sambanova": "https://api.sambanova.ai/v1",
    }
    return openai.OpenAI(base_url=bases[vendor], api_key=key)

Differences appear in header hints. Cerebras honors X-Rate-Limit aggressively and returns structured 429 bodies. SambaNova forwards cache-control for prefix caching, letting you pin a system prompt in silicon. Groq expects you to manage your own batching and does not expose cache TTLs. None support Anthropic-style tool schemas natively, but you can shove JSON into the prompt and parse it yourself.

Error handling is uniform enough for a wrapper:

try:
    resp = client.chat.completions.create(model="llama3-70b", messages=[])
except openai.RateLimitError as e:
    # backoff or route to another vendor
    pass

Ecosystem and Tooling

Groq has the loudest community. LangChain, LlamaIndex, and Vercel AI SDK all ship first-class Groq connectors. Cerebras is catching up, with official Python and JS SDKs and a growing Hugging Face integration. SambaNova’s ecosystem is enterprise-oriented: Kubernetes operators, on-prem appliances, and partner integrations with Snowflake and Databricks.

For local development, Groq’s rate limits are the most forgiving on free tiers (documented at 30 requests/minute for many models). Cerebras requires a waitlist for high throughput. SambaNova pushes you toward sales for production quotas.

Hard Limits and Quotas

Groq enforces per-minute token caps that scale with tier; exceeding them returns 429 with Retry-After. Cerebras imposes concurrent stream limits (often 10–20 on self-serve). SambaNova enforces max model size per account and may queue training jobs.

All three cap single-request output at 4K–8K tokens. If you need long completions, loop with continuation prompts. None permit indefinite streaming beyond their timeout window (typically 30–60 seconds per request), so design for chunked generation.

Head-to-Head Comparison

Which to Choose

Real-time chat UX where every millisecond counts. Pick Groq. The tokens per second Groq Cerebras SambaNova lead is consistent for interactive 70B models, and the free tier lets you prototype today.

High-volume batch generation with mixed model sizes. Cerebras wins when you can use 8B/13B models and want maximum tokens per second per dollar. Its batch mode handles offline dumps cleanly.

You need to adapt a model, not just call one. SambaNova is the only one of the three that lets you fine-tune on the same accelerated fabric. If your roadmap includes custom weights, accept the throughput tax and stay on SambaNova.

You want zero vendor lock-in. Use an OpenAI-compatible gateway that routes to all three. Set routing directives by latency SLA, and let fallback cover outages. That keeps the tokens per second Groq Cerebras SambaNova spread working for you instead of against you.

Cost-sensitive, fixed throughput. Groq and Cerebras are close; benchmark your exact prompt shape. SambaNova only makes sense if training is in scope.

The silicon is fast. The bottleneck is now your own orchestration code.