LLM cost per token vs speed: flagship models ranked

Choosing between frontier systems means weighing flagship models cost per token vs speed against raw capability. The gap between a $3/M input model that streams quickly and a $15/M model with half the throughput decides your infra bill and UX more than leaderboard points do. This ranking uses public list pricing (mid-2024) and observed latency characteristics from independent benchmark aggregators and production anecdotes.

Methodology

We scored each model on two axes: published token pricing (input/output per million tokens) and relative inference speed (time-to-first-token and sustained output throughput on hosted APIs). Self-hosted open weights are priced as effective compute cost or marked “variable.” Speed figures are intentionally qualitative—“low latency” vs “moderate”—because absolute numbers shift with batch size, region, and provider load.

A single OpenAI-compatible client call is enough to reproduce these measurements. The snippet below streams completions and logs timing:

from openai import OpenAI
import time

client = OpenAI()  # or point base_url at a gateway

def bench(model, prompt, max_tokens=200):
    t0 = time.time()
    chunks = []
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        stream=True,
    )
    first_token = None
    for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token is None:
                first_token = time.time() - t0
            chunks.append(chunk.choices[0].delta.content)
    total = time.time() - t0
    toks = sum(len(c.split()) for c in chunks)  # rough
    print(f"{model}: TTFT={first_token:.2f}s, {toks/total:.1f} tok/s")

Pricing sourced from provider websites; treat as baseline. A gateway such as n4n.ai forwards provider cache-control hints and meters per-token usage, so the same code above works across all models below without client changes.

1. Claude 3.5 Sonnet

Anthropic’s Claude 3.5 Sonnet sets the bar for price-performance among closed flagships. Public list pricing is $3 per million input tokens and $15 per million output tokens—identical output cost to GPT-4o but cheaper inputs. Independent latency measurements consistently place it at the top for time-to-first-token among non-streaming-optimized models, typically feeling snappy in chat UIs.

The model handles 200K context and exhibits strong tool-use reliability. For engineering teams building agents, the combination of low input cost and fast prefill means long system prompts don’t punish you twice: once in latency, once in price. If you need vision, it’s multimodal at no extra token cost.

Trade-off: output token price is still premium. For high-volume generation (summaries, transcripts), the $15/M output adds up. But the speed often offsets retry overhead.

2. GPT-4o

OpenAI’s GPT-4o is the default incumbent. Pricing is $5/M input, $15/M output. Speed is excellent—low TTFT and high sustained throughput on the official API, with solid multimodal support. It remains the safest choice for ecosystem compatibility: every SDK, proxy, and eval harness assumes it works.

Where it loses to Claude 3.5 Sonnet is input cost. If your workload is prompt-heavy (RAG, long instructions), you pay a 66% premium on input tokens for comparable latency. Output speed is roughly on par, so the decision comes down to whether you trust Anthropic’s safety tuning less than OpenAI’s.

For production routing, GPT-4o’s widespread availability means fallback is easy. When a provider is rate-limited, a gateway that honors client routing directives can shift traffic to Sonnet without code changes.

3. Gemini 1.5 Pro

Google’s Gemini 1.5 Pro is the long-context specialist. Pricing scales with context: $3.50/M input and $10.50/M output for prompts under 128K; above that, rates double. Speed is moderate—prefill on 1M-token inputs is slower than Claude or GPT-4o, but output streaming is competitive once generation starts.

The killer feature is the 1M–2M token window at reasonable cost. If your flagship models cost per token vs speed analysis includes processing entire codebases or books, Gemini wins on input economics despite slower warm-up. For short-chat workloads, the latency penalty isn’t worth the slightly lower output price.

Be aware of regional quota differences. In our tests, TTFT from us-central1 was fine; from other regions it degraded.

4. Llama 3.1 405B

Meta’s Llama 3.1 405B is the open-weight flagship. Hosted pricing varies: Together AI and others list roughly $5–$10/M output depending on batch. Self-hosted, you pay for eight H100s or equivalent—effectively a fixed cost that breaks even only at very high volume. Speed on self-hosted rigs depends entirely on tensor parallelism; a single node struggles to hit 20 tok/s, while a 4-GPU setup rivals API latency.

The advantage is data sovereignty and no per-token meter. For regulated workloads, that outweighs raw speed. The model’s quality is close to Claude 3.5 Sonnet on many coding tasks. But you own the ops burden: autoscaling, quantization, KV-cache management.

If you use a hosted endpoint, treat it like any closed model but expect more variance in TTFT because providers pack GPUs differently.

5. Mistral Large 2

Mistral Large 2 is the European challenger. API pricing is $2/M input, $6/M output—the cheapest flagship on this list by a wide margin. Speed is good: low TTFT, solid throughput, and a 128K context. It supports function calling and multilingual tasks competently.

The catch is capability ceiling. It trails Claude 3.5 Sonnet and GPT-4o on complex reasoning and agentic loops, but for extraction, translation, and moderate RAG it’s the best token economy. If your flagship models cost per token vs speed matrix is dominated by cost, this is the pick.

We’ve seen it degrade under heavy concurrent load on the reference API, so build retry logic.

6. Command R+

Cohere’s Command R+ targets enterprise RAG. Pricing is $3/M input, $15/M output. Speed is moderate—optimized for grounded generation rather than chat race conditions. Its differentiator is native RAG tooling and citation, which can save you tokens by reducing prompt scaffolding.

If your pipeline is retrieval-heavy, the built-in reranking and citation may offset the premium output price. Otherwise, Claude 3.5 Sonnet beats it on both speed and price-to-quality.

Summary Table

Synthesis

For most production systems, Claude 3.5 Sonnet is the pragmatic winner in the flagship models cost per token vs speed tradeoff: cheap inputs, fast responses, long context. GPT-4o remains the compatibility fallback. Gemini earns its keep only when context length explodes. Open weights matter for sovereignty, not for saving money unless you’re at scale.

Route by workload, not by brand. A single OpenAI-compatible endpoint that supports fallback and honors cache hints lets you shift between these without rewriting your client—measure TTFT in your own region before committing.