Aider and Continue.dev: choosing an LLM API backend

Choosing an Aider Continue.dev LLM API backend forces a real architecture decision: do you point both tools at the same OpenAI-compatible endpoint, run a local model, or sit a gateway in front of multiple providers? The wrong call shows up as latency spikes in your IDE or silent context truncation in batch refactors. This guide walks a concrete path from requirement mapping to production config.

1. Profile what each tool actually sends

Aider opens a repository map and ships large system prompts plus whole-file diffs. A typical session sends 20–40k tokens of context before you edit a single line. Continue.dev fires frequent small chat and autocomplete requests—often 200–1k token prompts with low tolerance for latency.

Their traffic shapes differ: Aider favors throughput and big context; Continue favors p99 latency on short completions. If you force both through one model, you either overpay for Continue or starve Aider. Decide whether you need a single Aider Continue.dev LLM API backend or two tailored endpoints.

What to measure

Run Aider with --verbose --no-auto-commits for a day on a real repo. Log Continue’s requests via its debug panel (Cmd+Shift+P → “Continue: Toggle Debug Log”). Note average prompt tokens, max context observed, and requests per minute. Without this data you are guessing.

2. Choose a topology

Three viable setups:

Direct provider. Set OPENAI_API_BASE to OpenAI or use Anthropic’s official SDK. Simple, but you own rate-limit backoff and model routing.

Local via Ollama. Good for Continue autocomplete if you have a 16GB+ GPU. Aider struggles on 7B models for cross-file reasoning; 13B+ helps but still lags cloud.

Unified gateway. A single OpenAI-compatible endpoint that fronts many models. For example, n4n.ai exposes one endpoint across 240+ models and fails over automatically when a provider is rate-limited, which removes the need to write your own retry loop for the Aider Continue.dev LLM API backend.

Tradeoff: gateways add a network hop and may mask provider-specific headers. Direct gives raw control but multiplies YAML. Local cuts cost to electricity but binds you to GPU uptime.

3. Configure Aider

Aider reads OPENAI_API_BASE and OPENAI_API_KEY from env. Model names use a provider/model scheme. For a custom backend:

export OPENAI_API_BASE=https://api.example.com/v1
export OPENAI_API_KEY=sk-yourkey
aider --model openai/gpt-4o --verbose

Store these in .env at project root; Aider loads it. For Anthropic through an OpenAI shim, use anthropic/claude-3-5-sonnet. If your backend uses a different auth header, patch with --api-key-header (check Aider docs for current flag).

Pitfall: Aider caches repo map in memory; if your endpoint silently caps context at 32k, edits fail weirdly. Verify max_input_tokens from the model list endpoint:

curl https://api.example.com/v1/models -H "Authorization: Bearer $KEY" | jq '.data[] | select(.id=="gpt-4o")'

4. Configure Continue.dev

Continue stores models in ~/.continue/config.json. It accepts an apiBase per model and supports multiple simultaneous entries.

{
  "models": [
    {
      "title": "GPT-4o for chat",
      "provider": "openai",
      "model": "gpt-4o",
      "apiBase": "https://api.example.com/v1"
    },
    {
      "title": "Local autocomplete",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Local autocomplete",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b"
  },
  "contextProviders": [
    { "name": "code" },
    { "name": "docs" }
  ]
}

Continue sends cache_control hints on supported backends. If your Aider Continue.dev LLM API backend strips them, expect higher token bills on repeated repo context. Test by inspecting request headers in the debug log.

5. Implement fallback and routing

If you skipped the gateway, write a thin proxy. Minimal Python Flask example with streaming:

from flask import request, Response
import requests

UPSTREAMS = ["https://api.openai.com/v1", "https://api.anthropic.com/v1"]

def proxy():
    for base in UPSTREAMS:
        try:
            r = requests.post(base + request.path,
                              headers=request.headers,
                              json=request.json, timeout=30, stream=True)
            return Response(r.iter_content(chunk_size=1024),
                            status=r.status_code,
                            headers=dict(r.headers))
        except requests.Timeout:
            continue
    return Response('{"error":"all upstreams dead"}', status=503)

A gateway like n4n.ai already honors client routing directives and forwards provider cache-control hints, so you can skip this code. But understand the failure modes: cascading timeouts kill IDE responsiveness. Always set a 5s connect timeout on Continue’s side.

6. Track cost and latency

Per-token metering is non-negotiable. Query your backend’s usage endpoint if it exposes one. With a direct provider, pull billing CSV. With a gateway, read the x-usage response header.

curl -s https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}' \
  -D - | grep x-usage

Set a daily budget alert. Continue’s autocomplete can burn 50k tokens/hour if misconfigured with a cloud model. Aider’s repo map refresh every N files also adds steady drain.

7. Common pitfalls

• Context mismatch: Aider expects 128k; your endpoint enforces 16k. Diffs get truncated with no error.
• Streaming stalls: Continue’s UI waits on SSE. If your proxy buffers, the spinner hangs forever.
• Cache header stripping: Backend drops cache_control, reprocessing system prompts each call. Double your bill.
• Model name drift: gpt-4 vs gpt-4o across tools causes silent fallback to expensive default.
• Rate limit blindness: Aider retries aggressively; Continue silently degrades to no autocomplete.
• Key scoping: Using same key for both tools makes per-tool metering impossible. Mint separate keys.

8. Decision checklist

Use this ordered path:

1. Log traffic from both tools for one day on representative repos.
2. If p99 latency for Continue < 300ms matters, run a local 7B+ coder model for autocomplete only.
3. Point Aider at a strong remote model (Claude 3.5 Sonnet or GPT-4o class) with verified context window.
4. If you manage more than two providers, adopt a gateway that provides automatic fallback for the Aider Continue.dev LLM API backend.
5. Verify context windows and cache headers with a test request before trusting defaults.
6. Meter tokens per route; alert at 80% of budget. Separate keys per tool.
7. Re-evaluate monthly; model roster changes fast.

That is the whole setup. Wire it, watch logs, adjust.