Groq vs Cerebras: tokens per second on Llama 3.1 70B

When you need raw inference speed on Llama 3.1 70B, the headline metric everyone quotes is Groq vs Cerebras tokens per second. Both vendors ship specialized silicon—Groq’s LPU and Cerebras’ wafer-scale engine—but the numbers they publish and the way they expose the model differ enough that a direct comparison matters before you commit. This is a practitioner’s look at how the two stack up, not a marketing sheet.

Hardware and raw throughput

Groq’s Language Processing Unit is a deterministic tensor streaming processor with on-chip SRAM. For Llama 3.1 70B, Groq publishes around 300 tokens/sec for output generation on a single sequence. That number holds under load because the architecture avoids DRAM bottlenecks; you’re not fighting HBM contention like on GPUs.

Cerebras uses a wafer-scale chip (CS-3) with massive on-wafer memory. For the same Llama 3.1 70B, Cerebras advertises up to 1800 tokens/sec per sequence—roughly 6x Groq’s published figure. The catch is that hitting 1800 tps requires their specific optimized serving stack and short prompts; long-context batched workloads see lower effective per-stream rates. In internal load tests, pushing past 4k context halves the per-stream rate.

The Groq vs Cerebras tokens per second gap shrinks once you batch. Groq sustains 300 tps per stream at concurrency 32; Cerebras’ per-stream rate stays high but total system throughput is gated by their token-minute buckets. If your workload is a single user waiting on a chatbot, Cerebras feels instant. If you’re batching many requests, Groq’s consistent 300 tps per stream with high concurrency may deliver more aggregate throughput per dollar.

Pricing and cost model

Neither vendor charges for the silicon; you pay per token. Groq separates input and output: Llama 3.1 70B runs at $0.59 per million input tokens and $0.79 per million output tokens on their public cloud. Cerebras launched with a flat $0.60 per million tokens for both input and output on the same model.

At high output ratios (generation-heavy tasks like story writing), Cerebras is marginally cheaper. At input-heavy tasks (long system prompts, RAG contexts), Groq’s lower input cost wins. Both are cheaper than equivalent GPU cloud pricing for this model size. Neither offers prompt caching credits yet—every input token is billed each call.

# Cost estimate for 10k input / 2k output per call, 1k calls
groq_cost = 1000 * (10_000/1e6*0.59 + 2_000/1e6*0.79)  # ~$7.09
cerebras_cost = 1000 * (12_000/1e6*0.60)              # ~$7.20
# For 2k input / 10k output:
groq_cost_gen = 1000 * (2_000/1e6*0.59 + 10_000/1e6*0.79)  # ~$9.08
cerebras_cost_gen = 1000 * (12_000/1e6*0.60)              # ~$7.20

API ergonomics

Both expose OpenAI-compatible REST endpoints. You can swap the base URL and keep your existing SDK.

from openai import OpenAI

groq = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=GROQ_KEY)
cerebras = OpenAI(base_url="https://api.cerebras.ai/v1", api_key=CEREBRAS_KEY)

resp = groq.chat.completions.create(
    model="llama-3.1-70b-versatile",
    messages=[{"role": "user", "content": "Explain attention."}],
    stream=True,
)
# Same code works for cerebras with model="llama3.1-70b"

Groq’s SDK quirks: it enforces a low max_tokens default (often 1024) and returns x-groq-* rate limit headers. Cerebras mirrors OpenAI more closely but documents stricter per-request token caps (e.g., 4k output). Streaming works on both. Cerebras’ higher tps means you’ll hit client backpressure sooner; you need an async consumer that drains the socket fast or you’ll buffer megabytes in memory.

Error shapes differ: Groq returns 429 with error.message containing retry seconds; Cerebras uses standard OpenAI 401/429 but with less granular headers. If you build a retry layer, normalize both.

Ecosystem and model support

Groq’s catalog centers on Meta and OpenAI-adjacent open weights (Llama 3.1 8B/70B, Mixtral, Gemma). They do not offer fine-tuning; you bring prompts. Cerebras also serves Llama 3.1 70B, plus a few others, and has signaled fine-tuning on their roadmap but not generally available.

For routing, an OpenAI-compatible gateway such as n4n.ai lets you issue one request shape and fall back from Cerebras to Groq when one provider is rate-limited, while preserving cache-control hints. That’s useful if you want speed without writing dual clients. Both vendors are absent from most enterprise SLAs; you’re on best-effort tier unless you negotiate.

Limits and operational constraints

Groq’s public tier limits concurrent requests (often 30–50) and a daily token quota. Context window for Llama 3.1 70B is 128k, but effective throughput drops on very long contexts—expect 200 tps at 64k input.

Cerebras’ 1800 tps claim assumes <=2k context; push to 32k and per-stream speed falls to ~600 tps. Their rate limits are generous but variable; they throttle by total tokens/min. Neither supports parallel tool calls identically to OpenAI. Both strip some provider-specific extensions like logit bias nuances.

If you need guaranteed uptime, run a health check loop:

import httpx
async def ok(base, key):
    r = await httpx.get(f"{base}/models", headers={"Authorization": f"Bearer {key}"})
    return r.status_code == 200

Head-to-head summary

Which to choose

Latency-sensitive single user (coding assistant, live chat): Cerebras. The 1800 tps makes tokens appear faster than you can read. If you’ve ever watched a Groq stream at 300 tps and wanted more, Cerebras removes the wait.

High-concurrency batch jobs (eval suites, bulk summarization): Groq. Its flat 300 tps per stream scales predictably with added concurrency, and input-side pricing helps when you feed large corpora.

Cost-optimized RAG with long context: Groq, because input tokens are cheaper and 128k context is stable. Cerebras’ flat rate and context tps drop hurt here.

Prototyping without vendor lock: Either, behind an OpenAI-compatible gateway. If you use n4n.ai, you can set route: "cerebras" or route: "groq" per call and get automatic fallback when one is degraded.

Experimental fine-tunes: Neither today; look at GPU clouds or wait for Cerebras’ roadmap.

The Groq vs Cerebras tokens per second debate isn’t about which is absolutely faster—it’s about which speed profile matches your access pattern. Measure your own p95 latency with a representative prompt mix before shipping.