Route local models with LiteLLM: Qwen3 and Llama 4

LiteLLM has become the de facto router for teams running multiple model providers behind a single OpenAI-compatible endpoint. Most tutorials focus on cloud APIs, but the same routing logic — fallback, load balancing, cost-aware selection — applies to local models served via Ollama, vLLM, or TGI. This guide walks through wiring Qwen3 and Llama 4 into LiteLLM’s config, defining routing rules, and verifying the pipeline end to end.

Step 1: serve the models locally

Before LiteLLM can route anything, the models need to be running and reachable. The fastest path for both Qwen3 and Llama 4 is Ollama, which handles quantization, context windows, and GPU offloading without a Docker compose file.

Pull the models you want. For this guide we use the 8B instruct variants, which fit on a single 24 GB GPU with 4-bit quantization:

ollama pull qwen3:8b
ollama pull llama4:8b

Start Ollama if it isn’t already running:

ollama serve

Verify both models respond. The default Ollama port is 11434:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Reply with only the word pong",
  "stream": false
}' | jq -r .response

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama4:8b",
  "prompt": "Reply with only the word pong",
  "stream": false
}' | jq -r .response

Both should return pong (or similar). If you see connection refused, ensure OLLAMA_HOST=0.0.0.0:11434 is set before starting ollama serve.

Note: If you prefer vLLM for higher throughput, replace the Ollama base URL with your vLLM server (http://localhost:8000/v1) and use the model IDs vLLM reports (/v1/models). The LiteLLM config below works identically.

Step 2: create the LiteLLM config file

LiteLLM reads a YAML file that defines models, routing policies, and callbacks. Create config.yaml in your working directory:

general_settings:
  master_key: "sk-local-master-key"  # change in production
  drop_params: true                  # strip unsupported params per model

model_list:
  - model_name: qwen3-8b
    litellm_params:
      model: ollama/qwen3:8b
      api_base: http://host.docker.internal:11434
      custom_llm_provider: ollama
    model_info:
      max_tokens: 32768
      input_cost_per_token: 0
      output_cost_per_token: 0

  - model_name: llama4-8b
    litellm_params:
      model: ollama/llama4:8b
      api_base: http://host.docker.internal:11434
      custom_llm_provider: ollama
    model_info:
      max_tokens: 32768
      input_cost_per_token: 0
      output_cost_per_token: 0

router_settings:
  routing_strategy: "latency-based-routing"
  fallback_models:
    qwen3-8b: ["llama4-8b"]
    llama4-8b: ["qwen3-8b"]
  retries: 2
  timeout: 120

litellm_settings:
  callbacks: ["langfuse"]  # optional, see Step 5
  callback_args: {}

Key fields explained:

• model_name is the logical name clients will call (e.g., qwen3-8b).
• litellm_params.model uses the ollama/ prefix so LiteLLM knows to hit the Ollama /api/chat endpoint.
• api_base uses host.docker.internal because LiteLLM runs in a container while Ollama runs on the host. If you run LiteLLM natively, use http://localhost:11434.
• router_settings.routing_strategy: "latency-based-routing" sends each request to the model with the lowest recent p95 latency. Other options: simple-shuffle, cost-based-routing, or a custom router_callback.
• fallback_models defines the chain when the primary model errors or times out. Here each model falls back to the other.

Step 3: launch the LiteLLM proxy

Run the proxy in a container so the host.docker.internal DNS resolves correctly:

docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000

Wait a few seconds for the container to start, then verify the /health/liveliness endpoint:

curl -s http://localhost:4000/health/liveliness
# {"status": "healthy"}

List the models the proxy exposes:

curl -s http://localhost:4000/v1/models | jq '.data[].id'

Expected output:

qwen3-8b
llama4-8b

If the list is empty, check docker logs litellm-proxy for connection errors to Ollama.

Step 4: send routed requests

Clients now call the proxy exactly like the OpenAI API. The model field selects the logical model name from your config.

Basic chat completion:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 64,
    "temperature": 0.7
  }' | jq -r '.choices[0].message.content'

Streaming works the same way:

curl -N http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama4-8b",
    "messages": [{"role": "user", "content": "Count to five."}],
    "max_tokens": 64,
    "stream": true
  }'

Verify routing behavior

To see latency-based routing in action, send a burst of requests and watch the proxy logs:

for i in {1..10}; do
  curl -s -o /dev/null -w "%{http_code} " \
    -H "Authorization: Bearer sk-local-master-key" \
    -H "Content-Type: application/json" \
    -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"ping"}],"max_tokens":4}' \
    http://localhost:4000/v1/chat/completions
done
echo

Then check logs:

docker logs litellm-proxy --tail 50

You’ll see lines like:

INFO: LiteLLM Router: Routing request to model: llama4-8b (latency-based-routing)
INFO: LiteLLM Router: Fallback triggered for qwen3-8b -> llama4-8b

If you kill Ollama (pkill ollama) and retry, the fallback chain activates and the request succeeds against the remaining model.

Step 5: add observability (optional but recommended)

Local models still need usage tracking. LiteLLM supports callbacks for Langfuse, Datadog, Prometheus, and custom HTTP webhooks. The simplest production-grade setup is Langfuse self-hosted.

Add to config.yaml under litellm_settings:

litellm_settings:
  callbacks: ["langfuse"]
  callback_args:
    langfuse:
      public_key: "pk-lf-xxx"
      secret_key: "sk-lf-xxx"
      host: "http://host.docker.internal:3000"

Restart the proxy:

docker restart litellm-proxy

Now every request — including fallbacks — appears in Langfuse with model name, latency, token counts, and cost (zero for local, but the field exists for hybrid setups).

If you prefer Prometheus, enable the built-in exporter:

general_settings:
  enable_prometheus_metrics: true

Then scrape http://localhost:4000/metrics for litellm_request_duration_seconds, litellm_requests_total, and per-model counters.

Step 6: advanced routing — per-request directives

LiteLLM honors x-litellm-routing headers for dynamic control without config changes. Use cases:

• Force a specific model for a tenant
• Skip fallback for latency-critical paths
• Route by capability (e.g., only models with >128k context)

Example: force Llama 4 for this request, no fallback:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-master-key" \
  -H "Content-Type: application/json" \
  -H "x-litellm-routing: {\"model\":\"llama4-8b\",\"fallback\":false}" \
  -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"ignored"}],"max_tokens":16}'

The model in the JSON body is ignored when the routing header is present. This is useful for multi-tenant gateways where the router decides based on policy, not the client.

Step 7: run LiteLLM as a systemd service (production)

Containers are fine for dev. For a VM or bare-metal box, run the proxy under systemd so it restarts on crash and survives reboots.

Create /etc/systemd/system/litellm.service:

[Unit]
Description=LiteLLM Proxy
After=network.target ollama.service
Requires=ollama.service

[Service]
Type=simple
User=litellm
WorkingDirectory=/opt/litellm
ExecStart=/opt/litellm/.venv/bin/litellm --config /opt/litellm/config.yaml --port 4000
Restart=on-failure
RestartSec=5
Environment=OLLAMA_HOST=0.0.0.0:11434

[Install]
WantedBy=multi-user.target

Install in a virtual environment:

sudo useradd -r -s /bin/false litellm
sudo mkdir -p /opt/litellm
sudo chown litellm:litellm /opt/litellm
sudo -u litellm python3 -m venv /opt/litellm/.venv
sudo -u litellm /opt/litellm/.venv/bin/pip install 'litellm[proxy]'
sudo cp config.yaml /opt/litellm/
sudo systemctl daemon-reload
sudo systemctl enable --now litellm

Verify:

systemctl status litellm
curl -s http://localhost:4000/health/liveliness

Step 8: benchmark and tune

Local model latency varies wildly with quantization, context length, and GPU memory pressure. Run a quick benchmark to set realistic timeouts and decide whether you need multiple Ollama instances behind a load balancer.

# bench.py
import asyncio, time, httpx

async def bench(model: str, n: int = 20):
    url = "http://localhost:4000/v1/chat/completions"
    headers = {"Authorization": "Bearer sk-local-master-key", "Content-Type": "application/json"}
    payload = {"model": model, "messages": [{"role": "user", "content": "Hi"}], "max_tokens": 16}
    latencies = []
    async with httpx.AsyncClient(timeout=60) as client:
        for _ in range(n):
            t0 = time.perf_counter()
            r = await client.post(url, headers=headers, json=payload)
            r.raise_for_status()
            latencies.append(time.perf_counter() - t0)
    print(f"{model}: p50={sorted(latencies)[n//2]:.2f}s  p95={sorted(latencies)[int(n*0.95)]:.2f}s  max={max(latencies):.2f}s")

asyncio.run(bench("qwen3-8b"))
asyncio.run(bench("llama4-8b"))

Run it:

python3 bench.py

Typical output on an RTX 3090 (4-bit, 8B params):

qwen3-8b: p50=0.42s  p95=0.78s  max=1.12s
llama4-8b: p50=0.51s  p95=0.94s  max=1.35s

Use the p95 to set router_settings.timeout in config.yaml (add 20-30% headroom). If p95 exceeds 30s, consider:

• Smaller quantization (Q3_K_M → Q4_K_M)
• Splitting across two GPUs with OLLAMA_NUM_GPU_LAYERS
• Running a second Ollama instance on another machine and adding it to model_list with a different api_base

Verification checklist

Before declaring the setup done, confirm each item:

1. Both models respond directly via Ollama (curl to port 11434).
2. LiteLLM proxy starts without errors (docker logs or systemctl status).
3. /v1/models returns both logical model names.
4. Chat completions succeed for each model via the proxy.
5. Fallback works: stop one model in Ollama (ollama stop qwen3:8b), send a request for that model, verify the other model answers.
6. Routing strategy behaves: send 20+ requests, check logs show distribution or latency-based selection.
7. Observability captures data: Langfuse shows traces, or Prometheus metrics increment.
8. Timeouts are realistic: benchmark p95 < configured timeout.

Common pitfalls

What’s next

You now have a single endpoint that routes between Qwen3 and Llama 4, fails over automatically, and emits metrics. From here you can:

• Add a third model (e.g., deepseek-r1:8b for reasoning) and route by task type using a custom router_callback.
• Implement per-tenant budgets with the budget_id callback argument.
• Place n4n.ai in front if you need a managed control plane that also reaches 240+ cloud models — same OpenAI-compatible contract, automatic provider fallback, and per-token metering across local and remote.

The config file is the source of truth. Version it, CI-test it, and treat model routing like any other infrastructure component.