vLLM on A100 vs hosted API: time to first token

When you measure vLLM A100 vs API time to first token, you are comparing a self-managed inference server on dedicated silicon against a shared black-box endpoint. The raw number matters less than what drives it: batching policy, queue depth, cold starts, and network hops.

Capabilities

Self-hosted vLLM on A100

vLLM gives you the weights, the scheduler, and the VRAM. You pick the model revision, apply AWQ or GPTQ quantization, set max_num_seqs to tune throughput vs latency, and expose an OpenAI-compatible server. On a single A100-40GB you can serve a 7B–13B model at full fp16, or a 70B model at 4-bit quant. You own the upgrade cycle.

Hosted API

A hosted API locks you into its model catalog. You get GPT-4-class or Claude-class models you cannot run locally, but you sacrifice visibility into batching. The provider decides priority, concurrency, and whether your request shares a batch with a thousand others. For many teams that trade is worth it.

Price and Cost Model

A dedicated A100 instance carries fixed cost whether you send requests or not. On common cloud providers a single A100-40GB runs roughly $1–3 per hour depending on region and commitment. At 10% utilization that is still $0.10–0.30 per hour burned. vLLM amortizes that cost only when you drive sustained traffic.

Hosted APIs charge per token. A 7B self-hosted model generating 1M output tokens on an A100 might cost ~$0.02 in compute if fully utilized; the same tokens from a frontier API could be $10–$30. The crossover point is traffic volume and model size.

# Rough monthly cost sanity check
a100_hourly = 1.50
hours = 30 * 24
monthly = a100_hourly * hours  # $1080 fixed
# API: $2 per 1M output tokens, 50M tokens/month = $100

Latency and Throughput: the TTFT Core

The central metric in vLLM A100 vs API time to first token is not average latency but tail behavior under load. vLLM with continuous batching can return the first token for a 32-token prompt in 40–120 ms when the batch is empty. Push 64 concurrent streams and TTFT degrades as the scheduler waits for a free slot and prefill compute.

Hosted APIs add network RTT plus provider-side queue. A small prompt to a mid-tier API often shows 200–600 ms TTFT on a good day, spiking to multiple seconds during peak. You cannot inspect why.

Measure both the same way:

import time, openai

# Point at vLLM
client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# Point at hosted API
# client = openai.OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")

start = time.time()
stream = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Explain TCP fast open in one paragraph."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        ttft = (time.time() - start) * 1000
        print(f"TTFT: {ttft:.1f} ms")
        break

Run that against your own vLLM container and against the API you are evaluating. The shape of the curve under asyncio load tells you more than a single shot.

Throughput interaction

TTFT and tokens-per-second are coupled. vLLM lets you trade TTFT for aggregate throughput by raising max_num_seqs. APIs hide this; you just get rate-limited.

Ergonomics and Ops

Running vLLM means owning CUDA drivers, Docker images, GPU metrics, and autoscaling. A Kubernetes HorizontalPodAutoscaler on GPU is nontrivial. You also handle model downloads and checksum validation.

An API is a key and an endpoint. You still build retries and fallback, but you never page someone for an ECC error.

Ecosystem

vLLM speaks the OpenAI request format and plugs into LangChain, LlamaIndex, and Ray. You can front it with your own gateway to add routing.

A hosted gateway changes the equation when it aggregates providers. For example, n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and automatically falls back when a provider is rate-limited or degraded, forwarding cache-control hints. That narrows the reliability gap that single-vendor APIs have versus self-hosted, without you running the metal.

Hard Limits

Which to Choose

Prototyping and low volume

Use a hosted API. The vLLM A100 vs API time to first token difference is irrelevant when you send 100 requests a day. Pay per token and ship.

Latency-sensitive, predictable load

If you have a steady stream of sub-100 ms TTFT requirements and the model fits on one A100, self-host vLLM. You control the scheduler and avoid network variance.

Cost-sensitive at scale

Beyond roughly 50–100M tokens/month on smaller models, the A100 hourly cost beats per-token pricing. Run vLLM with quantization and continuous batching.

Need frontier models

If the task requires GPT-4-class reasoning, you cannot self-host equivalently. Use the API, or a gateway that routes to it and falls back.

Variable bursty traffic

APIs win on elasticity. Spinning A100s up and down has friction; APIs just bill you. A gateway with multi-provider fallback covers provider outages without your own metal.

Regulated or air-gapped data

vLLM on premises is the only option. The TTFT penalty of local hardware is acceptable when exfiltration is not.

Pick based on where your traffic sits on the utilization curve, not on a single benchmark number.