Is self-hosting Mistral Large actually faster than an API?

The question of self-hosted Mistral Large latency vs API is usually framed as a cost debate, but latency is the sharper edge. For teams shipping interactive features, a 200ms difference in time-to-first-token changes product feel. The thesis here is simple: self-hosting is not automatically faster, and for most real workloads the API path will beat your own GPUs on tail latency, cold starts, and burst handling.

What Mistral Large actually demands

Mistral Large is a 123B-parameter dense model. You cannot serve it on a single consumer GPU. A production deployment needs at least two H100 80GB cards with tensor parallelism, or four A100 80GB with careful sharding. That hardware floor defines your latency baseline before you tune a single kernel.

The model itself is not unusually slow relative to its size, but size is the whole story. Prefill (processing the prompt) is memory-bandwidth bound across all GPUs. Decode (generating tokens) is latency-bound per sequence. No software trick removes the physics: a 123B model moves a lot of weights per token.

First token latency is the product

Users judge LLM features by time-to-first-token (TTFT). A chatbot that waits three seconds before showing anything feels broken, even if it then streams fast. In the self-hosted Mistral Large latency vs API comparison, TTFT is where the two paths diverge most.

A managed API terminates your TLS, authenticates, routes to an internal fleet, and often batches your prefill with other tenants’ requests. That sounds like overhead, but providers amortize it across thousands of concurrent requests. Your self-hosted vLLM instance does the same prefill on fewer GPUs with no tenant mixing.

Here is a minimal client measurement that mirrors what you should run on both paths:

import time, openai

client = openai.OpenAI(base_url="https://your-endpoint/v1")
t0 = time.perf_counter()
stream = client.chat.completions.create(
    model="mistral-large",
    messages=[{"role": "user", "content": "Explain Raft consensus in two paragraphs."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft = time.perf_counter() - t0
        break
print(f"TTFT: {ttft:.3f}s")

Run that against your own vLLM server and against a hosted API. The numbers will vary, but the shape is predictable: the API hides prefill cost via global batching; your server exposes it directly.

Standing up the self-hosted path

A realistic self-hosted stack uses vLLM or TGI. With vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-Large-Instruct-2407 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9

This gives you an OpenAI-compatible endpoint on localhost. The server applies continuous batching, so concurrent requests share the GPU. Under light load, a single request gets the full decode bandwidth. Under load, requests queue in the scheduler.

That queue is the part nobody mentions in the latency pitch. Self-hosting trades the provider’s multi-tenant queue for your own single-node queue. If you have one model replica and a traffic spike, your TTFT grows linearly with backlog.

The API path’s real cost

The API is not free of latency. You pay a network round-trip to the provider region, plus TLS and auth. If you are in us-east-1 and the API frontends in eu-west-1, you add roughly 100ms of fiber alone. But managed fleets absorb load spikes because they scale replicas horizontally behind a load balancer.

Consider this JSON routing directive a client might send to a gateway:

{
  "model": "mistral-large",
  "route": {
    "prefer": ["local-vllm"],
    "fallback": ["api-provider"]
  },
  "cache_control": {"type": "ephemeral"}
}

A gateway such as n4n.ai honors these hints, forwarding cache-control to the provider and failing over when a replica is degraded. That adds one hop but can reduce effective latency by avoiding a dead endpoint. For direct API calls without a gateway, you still rely on the provider’s internal routing.

Measuring total throughput, not just TTFT

TTFT gets the attention; tokens-per-second (TPS) determines whether a long generation finishes before the user gives up. Self-hosted Mistral Large on 2x H100 will deliver decent single-stream TPS, but TPS collapses if you oversubscribe the GPU memory. The API provider sells you TPS as a service-level metric, not a thing you tune.

A fair benchmark loops 100 requests at varying concurrency:

import asyncio, time, openai

async def hit(client, sem, prompt):
    async with sem:
        t0 = time.perf_counter()
        resp = await client.chat.completions.create(
            model="mistral-large", messages=[{"role":"user","content":prompt}], stream=False
        )
        return time.perf_counter() - t0

async def main(concurrency):
    client = openai.AsyncOpenAI(base_url="https://your-endpoint/v1")
    sem = asyncio.Semaphore(concurrency)
    prompts = ["Summarize RFC 792 in a sentence."] * 100
    results = await asyncio.gather(*[hit(client, sem, p) for p in prompts])
    print(f"p95 latency @ {concurrency}: {sorted(results)[-5]:.3f}s")

asyncio.run(main(8))

Run this against self-hosted and API. At concurrency 1, self-hosted may win by a hair if the network hop is large. At concurrency 16, the API usually wins because its fleet expands.

When self-hosting actually beats the API

Self-hosting is faster in exactly three situations:

Dedicated hardware, steady load

If you have two H100s sitting idle 70% of the day, self-hosting removes the network round-trip and the provider’s tenant mixing. Your TTFT becomes just prefill time on local silicon.

Data residency or air-gap

Regulated workloads can’t send prompts to a public API. Latency is secondary, but local serving avoids outbound TLS entirely.

Custom speculative decoding

You can tune draft models, quantization, and batch sizes for your exact prompt distribution. Providers won’t do that for one tenant.

Outside those, the self-hosted Mistral Large latency vs API math favors the API. You become responsible for GPU driver updates, model weight downloads, and CUDA OOM crashes at 3am.

The gateway middle ground

You don’t have to pick one. A routing layer can send low-latency-sensitive traffic to a local replica and spill overflow to an API. The earlier JSON snippet shows the client intent; the gateway executes it.

This pattern works only if you measure the added hop. A poorly placed gateway in a different region than your users will negate the local replica’s advantage. Place the gateway adjacent to compute.

Ops overhead is latency too

Engineers forget that “time to mitigate” is part of latency. When the API returns 503, you retry or fall back. When your vLLM pod OOMs, you paginate logs, restart, and possibly lose warm weights. The cold start of a 123B model loading from disk to GPU takes tens of seconds. Keep that in the latency budget.

Decisive takeaway

Default to the API for Mistral Large unless you have committed GPU capacity and a traffic profile that saturates it. The self-hosted Mistral Large latency vs API debate is not about raw speed; it’s about who absorbs variance. The API absorbs variance across a fleet; you absorb it on a single node. For interactive products, variance is the enemy. Self-host when you have the utilization to justify the ops tax, not because you assume “my GPU is faster than the cloud.” It usually isn’t, once you count the queue.

If you do self-host, benchmark at your real concurrency, not concurrency 1. And if you route through a gateway, measure the hop. The fastest path is the one you actually measured under load.