Building a multi-model CLI in Python: GPT-5, Claude Opus 4.8, Gemini 3

Building a multi-model CLI in Python forces you to confront provider differences head-on: auth schemes, response shapes, and rate limits vary wildly. This guide walks through a concrete architecture that wraps GPT-5, Claude Opus 4.8, and Gemini 3 behind one command surface without leaking vendor specifics into your business logic.

1. Define a provider-agnostic interface

Start by specifying what your CLI actually needs: send a prompt, get text back, optionally stream tokens. Everything else is noise. If you skip this step, you will end up with if model.startswith("gpt") checks scattered across your codebase, and every new provider becomes a grep-and-patch exercise.

from abc import ABC, abstractmethod
from typing import Iterator

class LLMClient(ABC):
    @abstractmethod
    def complete(self, prompt: str, **kwargs) -> str:
        ...

    @abstractmethod
    def stream(self, prompt: str, **kwargs) -> Iterator[str]:
        ...

The interface is deliberately tiny. You can extend it later with count_tokens or embed, but resist the urge to add provider-specific parameters to the base method. Pass those through **kwargs and let the adapter decide.

2. Collapse providers with an OpenAI-compatible gateway

The fastest path to a working multi-model cli python is to route every call through one OpenAI-compatible endpoint. A gateway like n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models, which means the same openai client talks to GPT-5, Claude Opus 4.8, and Gemini 3 without separate SDKs.

from openai import OpenAI

class UnifiedClient(LLMClient):
    def __init__(self, base_url: str, api_key: str, model: str):
        self.client = OpenAI(base_url=base_url, api_key=api_key)
        self.model = model

    def complete(self, prompt: str, **kwargs) -> str:
        resp = self.client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            **kwargs,
        )
        return resp.choices[0].message.content

    def stream(self, prompt: str, **kwargs) -> Iterator[str]:
        stream = self.client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            stream=True,
            **kwargs,
        )
        for chunk in stream:
            if chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content

Tradeoff: you lose direct access to provider-specific features unless the gateway forwards them. Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints, so you can pass extra_headers={"cache_control": "epoch"} and it reaches the backend. You also get automatic fallback when a provider is rate-limited or degraded, and per-token usage metering comes back in the standard usage object.

3. Native adapters when you need full fidelity

If you need Claude’s extended thinking blocks or Gemini’s native multimodal inputs, write thin adapters. Keep them behind the same interface so the CLI never knows the difference.

import anthropic

class ClaudeClient(LLMClient):
    def __init__(self, api_key: str, model: str = "claude-opus-4-8"):
        self.client = anthropic.Anthropic(api_key=api_key)
        self.model = model

    def complete(self, prompt: str, **kwargs) -> str:
        msg = self.client.messages.create(
            model=self.model,
            max_tokens=kwargs.get("max_tokens", 1024),
            messages=[{"role": "user", "content": prompt}],
        )
        return msg.content[0].text

import google.generativeai as genai

class GeminiClient(LLMClient):
    def __init__(self, api_key: str, model: str = "gemini-3"):
        genai.configure(api_key=api_key)
        self.model = genai.GenerativeModel(model)

    def complete(self, prompt: str, **kwargs) -> str:
        resp = self.model.generate_content(prompt)
        return resp.text

These add dependencies and separate error hierarchies, but they remove the middleman and expose raw request fields. Use them only where the gateway path is insufficient.

4. Build the CLI surface

Use click for subcommands and model selection. Avoid argparse if you want nested options without boilerplate; click also gives you CliRunner for tests free.

import click
from functools import partial

@click.command()
@click.option("--model", default="gpt-5")
@click.option("--stream", is_flag=True)
@click.argument("prompt")
def main(model, stream, prompt):
    client = build_client(model)  # factory based on model name
    if stream:
        for tok in client.stream(prompt):
            click.echo(tok, nl=False)
        click.echo()
    else:
        click.echo(client.complete(prompt))

def build_client(model: str) -> LLMClient:
    if model in ("gpt-5", "claude-opus-4-8", "gemini-3") and GATEWAY_URL:
        return UnifiedClient(GATEWAY_URL, API_KEY, model)
    if model == "claude-opus-4-8":
        return ClaudeClient(ANTHROPIC_KEY)
    if model == "gemini-3":
        return GeminiClient(GOOGLE_KEY)
    return UnifiedClient(OPENAI_URL, OPENAI_KEY, model)

A factory keeps the CLI dumb. The multi-model cli python codebase stays testable because the client is injected or selected in one place. Add a --json flag early if you plan to script the output; humans and pipes want different formats.

5. Streaming and error handling

Streaming differs across providers: OpenAI sends SSE deltas, Anthropic uses its own event stream, Gemini returns an iterator. Your stream method hides that. Wrap network calls in try/except for the SDK’s base exception and convert to a CLI exit code.

from openai import APIError

def safe_stream(client, prompt):
    try:
        for tok in client.stream(prompt):
            yield tok
    except APIError as e:
        raise click.ClickException(f"Provider error: {e.status_code}") from e

Pitfall: never buffer a full stream into memory for long outputs. Yield directly to stdout. Another: if you intercept KeyboardInterrupt, flush the partial line and print a newline or your terminal will eat the next prompt.

6. Routing, fallback, and cache control

In production, a model will rate-limit. Implement a simple fallback chain: try primary, on 429 switch to secondary.

def with_fallback(prompt, primary, secondary):
    try:
        return primary.complete(prompt)
    except APIError as e:
        if e.status_code == 429:
            return secondary.complete(prompt)
        raise

If you use a gateway, automatic fallback when a provider is degraded may already be configured server-side; check your provider docs. Per-token usage metering lets you attribute cost per command invocation—log resp.usage if available, or emit a post-run summary line when not streaming.

Cache control matters for long system prompts. With an OpenAI-compatible gateway that forwards headers, set extra_headers={"cache_control": "ephemeral"} on the create call. Native Anthropic clients use a different cache_control block inside the message body; do not assume the two are interchangeable.

7. Configuration and secrets

Store keys in environment variables, never in the command string or in a checked-in config. Use python-dotenv for local dev.

from dotenv import load_dotenv
import os

load_dotenv()
OPENAI_KEY = os.getenv("OPENAI_API_KEY")
ANTHROPIC_KEY = os.getenv("ANTHROPIC_API_KEY")
GOOGLE_KEY = os.getenv("GOOGLE_API_KEY")
GATEWAY_URL = os.getenv("N4N_GATEWAY")  # optional

Add a --config flag to point at an alternate env file for CI. Print a redacted view of active configuration with --debug so operators can verify which model and endpoint are live.

8. Testing and common pitfalls

Mock the client, not the network. Define a FakeClient implementing LLMClient and run your CLI via click.testing.CliRunner.

class FakeClient(LLMClient):
    def complete(self, prompt, **kw):
        return "ok"
    def stream(self, prompt, **kw):
        yield "ok"

def test_main():
    runner = CliRunner()
    result = runner.invoke(main, ["--model", "fake", "hello"], obj={"client": FakeClient()})
    assert result.exit_code == 0

Pitfalls we hit shipping this

• Token mismatches: GPT-5 and Claude count tokens differently. Don’t assume max_tokens means the same across models; set it per adapter.
• Timeout defaults: SDKs default to 10s or 600s inconsistently. Set explicit timeout on the client constructor.
• Model name drift: claude-opus-4-8 vs claude-opus-4-8-2025-xx. Pin versions in your factory or read from env.
• Streaming interrupts: Ctrl-C leaves partial lines. Use click.echo with nl=False and flush, then print a newline on cleanup.
• Error shape leakage: Anthropic errors are not OpenAI errors. Catch a common Exception in the CLI layer and map to exit code 1, but log the original type for debugging.

A multi-model cli python tool is only as stable as its thinnest adapter. Invest in the interface, mock relentlessly, and keep provider specifics behind one factory.