Serving Llama 4 with vLLM and an OpenAI-compatible API

Llama 4’s release changes the calculus for self-hosted inference: the Scout and Maverick variants bring mixture-of-experts architecture to the Llama line, which means you can serve a 17B active-parameter model that punches above its weight class — if your inference stack handles MoE routing correctly. This guide walks through serving Llama 4 with vLLM behind an OpenAI-compatible API, from hardware planning to production hardening.

Hardware planning for MoE models

Llama 4 Scout (17B active, 109B total) and Maverick (17B active, 400B total) are mixture-of-experts models. Only a subset of experts activate per token, but all expert weights must reside in GPU memory. This breaks the usual “active params × 2 bytes” rule of thumb for BF16.

For BF16 serving:

• Scout: ~220 GB VRAM (all 109B params loaded)
• Maverick: ~800 GB VRAM (all 400B params loaded)

Quantization changes the math dramatically. With AWQ or GPTQ at 4-bit:

• Scout: ~55 GB VRAM — fits on 2× H100 (80 GB) or 4× A100 (40 GB)
• Maverick: ~200 GB VRAM — needs 3× H100 or 8× A100 (40 GB)

Pitfall: vLLM’s MoE support requires the model to be in a format it recognizes. The official Meta releases use a custom Llama4ForConditionalGeneration class. As of vLLM 0.6.3, you need a recent nightly or to patch the model config. Check the vLLM GitHub for the current support matrix before committing hardware.

Model preparation and quantization

Don’t quantize blindly. MoE models are more sensitive to quantization error than dense models because routing decisions amplify small weight perturbations. Test your specific workload.

# AWQ quantization (recommended for vLLM)
pip install autoawq
python -m awq.quantize \
  --model_path meta-llama/Llama-4-Scout-17B-16E \
  --quant_path ./Llama-4-Scout-17B-16E-AWQ \
  --w_bit 4 \
  --q_group_size 128 \
  --zero_point \
  --version GEMM

For GPTQ (slightly faster quantization, similar quality):

pip install auto-gptq
python -m auto_gptq \
  --model_path meta-llama/Llama-4-Scout-17B-16E \
  --quant_path ./Llama-4-Scout-17B-16E-GPTQ \
  --bits 4 \
  --group_size 128 \
  --desc_act False

Tradeoff: AWQ generally preserves MoE routing quality better than GPTQ at 4-bit. At 8-bit, the difference is negligible. If you have VRAM headroom, 8-bit AWQ is the safe choice.

VLLM server configuration

vLLM’s OpenAI-compatible server is a single command, but production deployments need explicit configuration. Create a serve.sh:

#!/bin/bash
set -euo pipefail

MODEL_PATH="./Llama-4-Scout-17B-16E-AWQ"
TP_SIZE=2          # tensor parallel across 2 GPUs
PP_SIZE=1          # pipeline parallel (keep 1 for MoE)
MAX_MODEL_LEN=131072  # Scout supports 128K context
GPU_MEM_UTIL=0.90
DTYPE="auto"       # respects quantization dtype

python -m vllm.entrypoints.openai.api_server \
  --model "$MODEL_PATH" \
  --tensor-parallel-size "$TP_SIZE" \
  --pipeline-parallel-size "$PP_SIZE" \
  --max-model-len "$MAX_MODEL_LEN" \
  --gpu-memory-utilization "$GPU_MEM_UTIL" \
  --dtype "$DTYPE" \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key "$VLLM_API_KEY" \
  --served-model-name llama-4-scout \
  --disable-log-requests \
  --enable-prefix-caching \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256

Key flags explained:

• --enable-prefix-caching: Critical for multi-turn conversations and RAG workloads. Llama 4’s 128K context makes this high-impact.
• --max-num-batched-tokens: Controls the prefill chunk size. Lower values reduce peak memory but increase latency. Tune per workload.
• --served-model-name: The model identifier clients see in /v1/models. Use a stable name; swap weights behind it without client changes.

Tensor parallel vs pipeline parallel for MoE

Use tensor parallel (TP) only for MoE models. Pipeline parallel (PP) splits layers across GPUs, but MoE routing requires all experts for a given layer to be on the same device — otherwise you add cross-GPU communication for every token’s routing decision. vLLM’s MoE implementation assumes TP.

If you need more GPUs than TP supports (typically 8), you’re into multi-node territory. That requires Ray or a similar orchestrator and is outside single-server scope.

Health checks and readiness

Don’t rely on the HTTP port alone. vLLM’s /health endpoint returns 200 before the model finishes loading. Add a readiness probe that actually generates:

# readiness_check.py
import os
import sys
import requests
import json

BASE_URL = os.getenv("VLLM_BASE_URL", "http://localhost:8000")
API_KEY = os.getenv("VLLM_API_KEY")

headers = {"Authorization": f"Bearer {API_KEY}"} if API_KEY else {}

try:
    # Check /v1/models first (fast)
    r = requests.get(f"{BASE_URL}/v1/models", headers=headers, timeout=5)
    r.raise_for_status()
    models = r.json()["data"]
    if not any(m["id"] == "llama-4-scout" for m in models):
        sys.exit("Model not listed in /v1/models")

    # Actual generation test (catches OOM, weight loading issues)
    payload = {
        "model": "llama-4-scout",
        "messages": [{"role": "user", "content": "ping"}],
        "max_tokens": 5,
        "temperature": 0
    }
    r = requests.post(f"{BASE_URL}/v1/chat/completions", headers=headers, json=payload, timeout=30)
    r.raise_for_status()
    print("READY")
except Exception as e:
    print(f"NOT READY: {e}")
    sys.exit(1)

Run this in a Kubernetes readiness probe or systemd ExecStartPre equivalent.

Client integration: OpenAI SDK compatibility

The vLLM server implements the OpenAI /v1/chat/completions and /v1/completions endpoints. Most OpenAI SDKs work unchanged:

from openai import OpenAI

client = OpenAI(
    base_url="http://your-vllm-host:8000/v1",
    api_key=os.getenv("VLLM_API_KEY")  # or your configured key
)

response = client.chat.completions.create(
    model="llama-4-scout",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain mixture-of-experts in two sentences."}
    ],
    temperature=0.7,
    max_tokens=512,
    extra_body={
        "repetition_penalty": 1.1,  # vLLM-specific extension
        "top_k": 50                 # vLLM-specific extension
    }
)
print(response.choices[0].message.content)

Pitfall: The extra_body parameters (repetition_penalty, top_k, min_p, ignore_eos) are vLLM extensions. They work but aren’t part of the OpenAI spec. If you swap to a different OpenAI-compatible gateway later, strip them or guard with feature detection.

Streaming and timeouts

Streaming is essential for UX. vLLM supports SSE streaming via stream: true. Configure your reverse proxy (nginx, Traefik, Envoy) with generous timeouts:

# nginx snippet
location /v1/chat/completions {
    proxy_pass http://vllm-backend;
    proxy_read_timeout 300s;        # long generations
    proxy_send_timeout 300s;
    proxy_buffering off;            # critical for SSE
    proxy_cache off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

Without proxy_buffering off, nginx buffers the entire response before forwarding — defeating streaming.

Observability: metrics and logging

vLLM exposes Prometheus metrics at /metrics. Key metrics to alert on:

Enable structured JSON logging in vLLM 0.6+:

export VLLM_LOGGING_LEVEL=INFO
export VLLM_LOGGING_CONFIG='{"version": 1, "formatters": {"json": {"format": "%(asctime)s %(levelname)s %(name)s %(message)s", "class": "pythonjsonlogger.jsonlogger.JsonFormatter"}}, "handlers": {"console": {"class": "logging.StreamHandler", "formatter": "json"}}, "root": {"handlers": ["console"], "level": "INFO"}}'

Common production pitfalls

1. Context length vs KV cache memory

Llama 4 Scout supports 128K context, but KV cache scales linearly with sequence length. At 4-bit quantization with TP=2:

• 128K context ≈ 8 GB KV cache per sequence (BF16 would be 16 GB)
• With max_num_seqs=256, worst-case KV cache = 2 TB — impossible

Fix: Set max_model_len to your actual maximum (e.g., 32K for RAG, 8K for chat). Use max_num_batched_tokens to limit prefill memory. Monitor gpu_cache_usage_perc and scale max_num_seqs down if it stays high.

2. MoE expert imbalance

Some experts see disproportionate traffic, causing GPU compute imbalance across TP ranks. vLLM 0.6+ includes expert parallelism (EP) support, but it’s experimental. For now, accept slight imbalance or reduce TP size.

3. Flash attention and sliding window

Llama 4 uses sliding window attention in some layers. vLLM’s flash attention kernel handles this, but only if max_model_len ≤ the kernel’s supported window (typically 16K–32K depending on GPU). For longer contexts, vLLM falls back to a slower kernel. Test your latency at target context lengths.

4. Weight loading time

A 55 GB AWQ model takes 60–120 seconds to load on 2× H100. Your orchestration must account for this. Use a pre-warm step in your deployment pipeline, or keep a warm standby.

Scaling beyond a single node

When single-node TP hits limits, you have two paths:

Option A: Data parallel replicas — Run multiple vLLM servers behind a load balancer. Each serves independently. Simple, but no request-level sharing of KV cache.

Option B: Disaggregated prefill/decode — vLLM’s experimental disaggregated serving separates prefill (compute-heavy) from decode (memory-heavy). This improves throughput for long-context workloads but adds operational complexity. Not recommended until vLLM 0.7+ stabilizes it.

For most teams, Option A with a smart router (like n4n.ai’s gateway, which handles per-token metering and automatic fallback across replicas) is the pragmatic choice.

Checklist before going live

• Quantization validated on your eval set (not just perplexity — test routing quality)
• max_model_len set to actual requirement, not model maximum
• max_num_seqs and max_num_batched_tokens tuned for your GPU memory
• Readiness probe does a real generation, not just port check
• Reverse proxy configured for SSE streaming (buffering off, long timeouts)
• Prometheus scraping /metrics with alerts on queue depth and cache usage
• Structured JSON logging enabled and aggregated
• Load test at 2× expected peak traffic with realistic prompt/completion lengths
• Rollback plan: previous model weights tagged, deployment reversible in < 5 min

Closing thought

Serving Llama 4 with vLLM is straightforward once you respect the MoE memory profile. The OpenAI-compatible API means your application code barely changes — but the operational surface area (quantization validation, KV cache budgeting, MoE routing quirks) is where production stability lives. Start with Scout on 2× H100 at 4-bit AWQ, instrument heavily, and scale from evidence.