Automatic model fallback in LiteLLM to cut GPT-4 spend

Most teams pin their critical paths to GPT-4 and eat the bill. With litellm automatic model fallback cost savings become real: you define a model chain, and the proxy retries on cheaper or available models when the primary fails or hits a rate limit. This guide walks through a production-grade setup that cuts GPT-4 spend without sacrificing output quality for most traffic.

Step 1: Install and run the LiteLLM proxy

Running your own inference gateway sounds heavy until you see the 50 lines it takes. LiteLLM’s proxy is a FastAPI app that translates the OpenAI API shape to 100+ backends and adds routing logic.

Install the proxy extra and boot a local gateway. You need Python 3.9+.

pip install 'litellm[proxy]'
export OPENAI_API_KEY="sk-..."
export AZURE_API_KEY="..."

Create a minimal config.yaml. We’ll expand it in the next step.

touch config.yaml
litellm --config config.yaml --port 4000

The proxy now listens on http://localhost:4000 and speaks the OpenAI API shape. Auth is terminated at the proxy; your app sends any dummy key, and the proxy injects the real provider credentials from environment or config. For production, front it with TLS and a real auth layer (the proxy supports virtual keys).

Step 2: Declare your model chain with fallbacks

The core of litellm automatic model fallback cost savings is the router_settings.fallbacks map. You list a primary model and an ordered list of backups. The proxy tries them in sequence on failure.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: os.environ/AZURE_API_KEY
      rpm: 10 # low quota to trigger fallback fast in testing
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: os.environ/OPENAI_API_KEY
  - model_name: mistral-7b
    litellm_params:
      model: ollama/mistral-7b
      api_base: http://localhost:11434

router_settings:
  fallbacks: [{"gpt-4": ["gpt-3.5-turbo", "mistral-7b"]}]
  num_retries: 2
  timeout: 30

The fallbacks entry says: if gpt-4 raises a retryable error (429, 500, timeout), route to gpt-3.5-turbo, then to a local mistral-7b. Order matters—put the cheapest acceptable model last. The model_name is an alias your app uses; the litellm_params.model is the real provider path. You can map many aliases to the same backend for A/B or gradual rollout.

If you run multiple Azure deployments, add them as separate entries with the same model_name and LiteLLM load-balances across them before falling back. That shields you from a single region outage.

Step 3: Tune fallback triggers

A hard 429 is not the only signal. LiteLLM retries on status codes you whitelist and on latency breaches.

router_settings:
  fallbacks: [{"gpt-4": ["gpt-3.5-turbo"]}]
  num_retries: 3
  timeout: 20
  retry_on_status_codes: [429, 500, 502, 503]
  allowed_fails: 1 # bail after one provider failure

Set allowed_fails: 1 so a single degraded provider doesn’t burn user latency. The litellm automatic model fallback cost savings come from shedding GPT-4 calls that would otherwise queue or error. But beware: if you set timeout too low, you’ll fallback on slow-but-valid GPT-4 responses. Measure p95 latency on your Azure instance first; set timeout to p95 + 2s.

Step 4: Call the proxy from your app

Point the OpenAI client at the proxy. No application code changes beyond the base URL.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-1234"  # any non-empty string; proxy handles auth upstream
)

resp = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Summarize: LiteLLM routes models."}],
    temperature=0.2,
)
print(resp.model)  # prints gpt-4, gpt-3.5-turbo, or mistral-7b depending on path

When Azure GPT-4 is healthy, you get GPT-4. When it 429s, the proxy returns gpt-3.5-turbo without your code noticing beyond the model field. This is where litellm automatic model fallback cost savings show up in practice: the same model="gpt-4" request silently costs an order of magnitude less on fallback.

Streaming works identically:

stream = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Draft a SQL query"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

The fallback happens before the first token, so streaming clients see no interruption.

Step 5: Force a GPT-4 outage to verify fallback

Break the primary on purpose. Edit config.yaml to use a bogus key for gpt-4, restart, and send one request.

  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_key: "bad-key"

litellm --config config.yaml --port 4000 &
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"ping"}]}'

You should receive a 200 with "model":"gpt-3.5-turbo". The proxy logs show:

LiteLLM: AzureException 401, falling back to gpt-3.5-turbo

Revert the key after the test. For a cleaner test, keep the valid key but set rpm: 0 on gpt-4; LiteLLM treats zero quota as immediate fallback.

Step 6: Measure litellm automatic model fallback cost savings

LiteLLM tracks per-model token usage. Query the proxy’s spend endpoint to see how many requests escaped GPT-4.

curl http://localhost:4000/spend \
  -H "Authorization: Bearer sk-1234"

Response includes spend_per_model. If gpt-3.5-turbo shows nonzero while gpt-4 is lower than your pre-fallback baseline, the chain works. For durable metrics, scrape /metrics with Prometheus and graph fallback rate:

# prometheus.yml snippet
scrape_configs:
  - job_name: litellm
    static_configs:
      - targets: ["localhost:4000"]

Alert when fallback rate exceeds 20%—that signals a primary provider outage, not normal cost tuning. To put numbers on it, multiply escaped token counts by public list prices: GPT-4 pricing is an order of magnitude above GPT-3.5-turbo at public rates, so every 1M tokens shifted cuts spend roughly 90%. The litellm automatic model fallback cost savings are therefore directly proportional to your fallback rate.

Step 7: Add guardrails before production

Fallback is not free. GPT-3.5-turbo drops instruction following on complex JSON. Mitigate:

• Cache identical prompts. Set cache: true in router settings to short-circuit repeats on the cheap model.
• Eval a sample. Run 100 production prompts through both models, diff outputs with a judge LLM.
• Honor routing hints. LiteLLM passes metadata to providers; you can force gpt-4 for high-value users by skipping fallback via litellm_params: {"force_fallback": false} per key.

router_settings:
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379

Also set model_group aliases so you can later swap the fallback chain without app changes. For example, map premium to the GPT-4 chain and standard to a 3.5-only chain, then shift traffic at the DNS level.

Managed alternative

If operating the proxy and Redis adds surface area you don’t want, n4n.ai exposes one OpenAI-compatible endpoint that applies automatic fallback when a provider is rate-limited or degraded, spanning 240+ models behind a single URL with per-token metering. The routing logic is equivalent to the config above, but you skip the YAML.

Verify success checklist

• Proxy boots with config.yaml and /v1/models lists gpt-4.
• A bad key on gpt-4 returns a valid completion from gpt-3.5-turbo.
• /spend shows tokens attributed to fallback models.
• Fallback rate metric is scraped and alerting is armed.
• Cache hit rate >0 on repeated traffic.
• Streaming request completes with fallback model when primary is down.

That’s the full loop. You now pay GPT-4 rates only when GPT-4 actually answers.