When self-hosting beats an API: a latency breakeven analysis

Most teams pick a hosted API because it’s the path of least resistance, but once you cross a certain request volume, the self-hosted LLM vs API latency breakeven flips in favor of owning your inference stack. That point isn’t just about dollars per token—it’s about tail latency under your specific concurrency pattern and whether you can amortize GPU idle time across enough traffic.

1. Define your latency budget and workload shape

Before benchmarking anything, write down the number that matters: the maximum p95 latency your product can tolerate. A chat UI might survive 800 ms; a synchronous code-completion hook in an IDE probably can’t exceed 200 ms.

Then characterize the load:

• Average input/output tokens per request
• Requests per second (RPS) at peak
• Burst factor (peak / average)
• Whether requests are independent or need shared context

A batch-friendly workload (many independent short prompts) favors self-hosting because continuous batching hides per-request overhead. A spiky, low-volume workload favors APIs because your GPUs would sit idle 90% of the time.

Measure your current traffic with a tiny sampler:

import asyncio, time, httpx

async def measure(client, payload, n=100):
    samples = []
    for _ in range(n):
        t0 = time.perf_counter()
        await client.post("https://api.example.com/v1/chat/completions", json=payload)
        samples.append(time.perf_counter() - t0)
    samples.sort()
    p50 = samples[len(samples)//2]
    p95 = samples[int(len(samples)*0.95)]
    return p50, p95

# Run against production-like traffic later

If you don’t know these numbers, stop. You cannot compute a breakeven without them.

2. Measure API baseline honestly

API latency in docs is best-case. You care about what happens when the provider throttles you or a region degrades. Sample across a full day, not five minutes.

If you want to avoid writing bespoke retry and fallback code while sampling multiple backends, a gateway like n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is degraded, which lets you collect comparable p95 numbers in an afternoon.

A minimal measurement loop:

curl -s -o /dev/null -w "%{time_total}\n" \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' \
  https://api.example.com/v1/chat/completions

Wrap that in a shell loop with timestamps and plot the distribution. Record:

• p50, p95, p99
• token throughput (tokens / latency) for your real prompt size
• error rate under load

Common pitfall: benchmarking with a 5-token prompt then shipping 2k-token RAG contexts. Time-to-first-token scales with input processing; measure both TTFT and inter-token latency.

3. Model the self-hosted path

Pick a reference server. vLLM or TensorRT-LLM are the pragmatic choices for throughput today. A single A100 80GB runs a 70B model in FP16 with tensor parallelism across two GPUs, or a 7B/13B model with massive batch headroom.

Launch a vLLM instance:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 256

Then hit it with the same client code from step 1. The key metrics:

• Max sustained RPS before p95 degrades >20%
• KV cache hit behavior under your context lengths
• GPU utilization at that RPS (idle cost matters)

Self-hosting trades network round-trip for compute queueing. If your API p95 is 600 ms because of transit + provider queue, and local p95 is 120 ms but drops to 900 ms when you exceed batch capacity, the breakeven is a function of how often you exceed that capacity.

4. Compute the breakeven

The self-hosted LLM vs API latency breakeven has two axes: cost and latency. They intersect when self-hosted latency is acceptable and its amortized cost per request drops below API cost.

Cost breakeven formula:

daily_gpu_cost = gpu_price_per_hour * 24
api_daily_cost = rps * 86400 * avg_tokens * api_price_per_token
breakeven_rps = daily_gpu_cost / (86400 * avg_tokens * api_price_per_token)

Latency breakeven is fuzzier. If local p95 stays under your budget up to max_rps, and your observed peak RPS exceeds breakeven_rps, self-hosting wins on both axes.

def breakeven_rps(gpu_cost_hr, avg_tokens, api_cost_per_token):
    day_cost = gpu_cost_hr * 24
    return day_cost / (86400 * avg_tokens * api_cost_per_token)

# Example with real public numbers: A100 ~$1.5/hr on commodity cloud, API ~$0.10/1M tokens
print(breakeven_rps(1.5, 1500, 0.10/1_000_000))  # ~23 RPS sustained

That 23 RPS is a real crossing point for many 7B-class deployments. Below it, the GPU sits idle and the API is cheaper. Above it, you pay for metal whether you use it or not, so utilization is your only lever.

5. Run a shadow test

Modeling lies. Shadow testing doesn’t.

Mirror production traffic to both paths. Log latency, token counts, and output diffs (semantic similarity, not exact match). Run for at least one full weekly cycle.

async def shadow(client_api, client_local, req):
    api_task = client_api.post("/v1/chat/completions", json=req)
    local_task = client_local.post("/v1/chat/completions", json=req)
    api_res, local_res = await asyncio.gather(api_task, local_task)
    log(latency(api_res), latency(local_res), req["id"])

Pitfalls exposed here:

• Cold start on local after autoscaler shrinks pool
• Quantization drift (INT8/FP8 changes logits slightly)
• Provider API version silently changes system prompt handling

If local p95 is within 10% of API p95 and cost model shows savings, proceed.

6. Common pitfalls and tradeoffs

KV cache fragmentation. Long contexts with variable lengths waste memory. Use --max-num-seqs tuned to your trace, not defaults.

Multi-tenant interference. If you share GPUs across teams, your p95 becomes someone else’s batch job. Dedicated pools fix it but raise cost.

Model swaps. Hosting three models means either three server groups or a router with loading latency. The self-hosted LLM vs API latency breakeven gets worse when you spread volume across many models.

Provisioning lag. APIs scale instantly; your Kubernetes GPU node might take 5 minutes to appear. Keep a warm minimal pool.

Compliance theater. Self-hosting doesn’t automatically make you compliant. You still need logging, redaction, and patch cadence.

7. Decision checklist

An ordered path you can execute this quarter:

1. Instrument production to extract p50/p95 latency, token counts, and RPS distribution.
2. Baseline two API providers plus a gateway fallback over 7 days.
3. Stand up one vLLM node for your dominant model class; load-test with k6 or locust.
4. Calculate cost breakeven RPS using real GPU quotes and API price sheets.
5. Shadow 100% of traffic for one week; compare latency histograms and output quality.
6. Threshold on utilization: if peak RPS > breakeven and local p95 < budget, migrate read-path traffic first.
7. Keep the API as overflow. Even self-hosters need a escape hatch for spikes.

The self-hosted LLM vs API latency breakeven isn’t a myth. It’s a number you can derive, test, and watch drift as your product scales. Treat it as a living metric, not a one-time memo.