Codestral speed benchmark for coding tasks

Any serious Codestral speed benchmark for coding tasks has to separate time-to-first-token from sustained generation throughput, because interactive coding assistants live and die on both. Codestral 22B, Mistral’s code-specialized model, sits in a niche where those numbers matter more than leaderboard accuracy alone.

Why code completion is latency-bound

A developer waits on the model. If a completion takes two seconds to start and then dribbles out at five tokens per second, the experience feels broken regardless of correctness. Autocomplete and inline suggestions demand sub-second TTFT and enough generation speed to finish a line before the programmer moves on.

Batch processing of repo-level tasks is different. There, total throughput per dollar wins, and a slower model that saturates a GPU queue may be cheaper. A Codestral speed benchmark must capture both ends.

What Codestral 22B is

Codestral is a 22-billion-parameter dense transformer trained specifically on code and natural language for code. It supports fill-in-the-middle (FIM) via a prefix/suffix training objective and handles 80+ programming languages with a 32k context window.

It is not a mixture-of-experts model. Every token engages all 22B parameters, so inference cost scales predictably with batch size and sequence length. That makes it easier to reason about than sparse models, but it also means you cannot skip inactive experts for speed.

Designing a fair Codestral speed benchmark

Metrics that matter

Track two numbers:

• TTFT: milliseconds from request send to first generated token.
• Generation throughput: completed tokens divided by generation time (excluding TTFT).

Report them separately. A gateway can stream quickly but queue requests, inflating TTFT. Use usage from the API response for exact token counts when streaming is disabled, or count deltas when streaming.

Workload shape

Use real prompts. For FIM, structure the request as:

{
  "model": "codestral-22b",
  "prompt": "<s>def add(a, b):\n    [SUFFIX]\n    return a - b\n"
}

That prefix/suffix format triggers the trained infill behavior. For instruction mode, wrap in the standard chat template. Do not mix the two in the same benchmark run; FIM avoids the instruction overhead but uses different attention patterns.

Keep context near your production size. A 500-token prompt behaves differently from a 16k-token file. A Codestral speed benchmark on toy snippets will overestimate real-world performance.

Warm-up and variance

GPU kernels need warm-up. Discard the first three requests. Then take the median of at least ten runs; p99 TTFT is what users feel during traffic spikes.

Prefix caching changes the game. vLLM and similar servers cache identical prefix KV states. If your benchmark replays the same file header, you measure cache hits, not cold starts. State which mode you used.

A minimal benchmark harness

The script below measures TTFT and tokens/sec against any OpenAI-compatible endpoint. Point it at your local vLLM, a cloud provider, or a gateway.

import time, os, openai

client = openai.OpenAI(
    base_url=os.environ.get("OPENAI_BASE_URL", "https://api.n4n.ai/v1"),
    api_key=os.environ["OPENAI_API_KEY"]
)

def bench_codestral(prefix: str, suffix: str, max_tokens: int = 64):
    prompt = f"<s>{prefix}\n[SUFFIX]\n{suffix}\n"
    t0 = time.time()
    stream = client.completions.create(
        model="codestral-22b",
        prompt=prompt,
        max_tokens=max_tokens,
        temperature=0,
        stream=True
    )
    first_token_ts = None
    tokens = 0
    for chunk in stream:
        if chunk.choices[0].text:
            if first_token_ts is None:
                first_token_ts = time.time()
            tokens += 1
    gen_time = time.time() - first_token_ts
    ttft = (first_token_ts - t0) * 1000
    print(f"TTFT: {ttft:.0f}ms | {tokens} tokens in {gen_time:.2f}s "
          f"=> {tokens/gen_time:.1f} t/s")

bench_codestral(
    "def fib(n):\n    if n <= 1:\n        return n",
    "    return fib(n-1) + fib(n-2)"
)

Using an OpenAI-compatible gateway such as n4n.ai lets the same script run across 240+ models and honors client routing directives, so you can compare Codestral to alternatives without rewriting the client.

Example output shape:

TTFT: XXXms | NN tokens in Y.YYs => Z.Z t/s

Replace with your measured medians.

Interpreting a reference run

On a single 24GB GPU with INT4 quantization, Codestral 22B loads with headroom for a few concurrent sequences. TTFT stays under a few hundred milliseconds for short prompts; generation lands in double-digit tokens per second per sequence. That is enough for inline suggestions but not for streaming a whole file instantly.

FP16 doubles memory and halves achievable concurrency on the same hardware. Throughput per token is similar; the difference is how many independent requests fit before queuing.

These are qualitative observations from common serving setups, not vendor numbers. Your CPU, PCIe, and kernel version shift them.

Serving configuration that matters

A bare vllm serve call leaves performance on the table. Tune explicitly:

vllm serve mistralai/Codestral-22B-v0.1 \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching

Capping max context to 8k instead of 32k shrinks KV-cache footprint, letting you raise batch size. If your coding tasks rarely exceed a few thousand tokens, this directly improves both TTFT and throughput.

Tradeoffs in serving

Quantization

INT4 cuts VRAM to roughly 12–14GB for weights, leaving room for KV cache. Accuracy on code tasks degrades slightly but remains usable for autocomplete. INT8 is safer for edge-case correctness and still fits on 24GB with careful batching. Avoid FP16 on 24GB unless you accept a batch size of one.

Context length

Long contexts multiply KV-cache memory. A 32k context with a large batch will OOM a 24GB card long before compute is the limit. Cap max context in the serving config to match your real needs.

Concurrency

Codestral is dense, so compute per token is fixed. Increase batch size only until KV cache or memory bandwidth saturates. Beyond that, requests queue and TTFT spikes. For interactive use, limit concurrency and reject or queue excess load.

Codestral vs smaller and larger models

A 7B code model answers faster and fits on cheaper hardware, but struggles with multi-file reasoning. A 70B general model produces better complex refactors but costs multiples of the latency and VRAM.

The Codestral speed benchmark for coding tasks shows the 22B sits between: noticeably smarter than 7B, noticeably cheaper than 70B. For most editor integrations, that band is the product sweet spot.

When to avoid Codestral

If you need guaranteed sub-100ms TTFT on a 16k context, no 22B model on commodity GPUs will hold that under load. Use a smaller distilled model or a cached suggestion system.

If you need document-level architecture reasoning, Codestral’s training focus on code snippets leaves gaps versus frontier models. Speed is irrelevant if the output is wrong.

Takeaway

Run your own Codestral speed benchmark with the harness above before trusting marketing claims. With INT4 quantization on a single 24GB GPU, Codestral 22B delivers interactive latency for code completion and reasonable throughput for small batches, making it the pragmatic default for self-hosted coding assistants. Step up to larger models only when quality gaps hurt, and step down only when hardware forces you.