Testing an LLM CLI tool: mocking GPT-5 responses in Python

When you ship a command-line tool that talks to a frontier model, you need a repeatable strategy for testing llm cli mock gpt-5 python without expending real tokens or flaking on network latency. This guide builds a local OpenAI-compatible mock server and a pytest harness that drives your CLI against it, so you can validate argument parsing, output formatting, and error paths offline.

Step 1: Scaffold a minimal CLI that calls GPT-5

Start with a thin wrapper around the OpenAI Python client. Keep the model name configurable but default to gpt-5. The key is to read the base URL from the environment so tests can redirect it without touching code paths.

# cli.py
import os
import argparse
from openai import OpenAI

def main():
    parser = argparse.ArgumentParser(description="Ask GPT-5 from the terminal")
    parser.add_argument("prompt", help="Prompt text")
    parser.add_argument("--model", default="gpt-5")
    parser.add_argument("--stream", action="store_true")
    args = parser.parse_args()

    client = OpenAI(
        api_key=os.environ.get("OPENAI_API_KEY", "dummy"),
        base_url=os.environ.get("OPENAI_BASE_URL", "https://api.openai.com/v1"),
    )
    resp = client.chat.completions.create(
        model=args.model,
        messages=[{"role": "user", "content": args.prompt}],
        temperature=0,
        stream=args.stream,
    )
    if args.stream:
        for chunk in resp:
            if chunk.choices[0].delta.content:
                print(chunk.choices[0].delta.content, end="", flush=True)
        print()
    else:
        print(resp.choices[0].message.content)

if __name__ == "__main__":
    main()

I prefer argparse over heavier frameworks for internal tools—less abstraction between you and sys.argv. The OPENAI_BASE_URL injection point is non-negotiable; without it you cannot run tests hermetically.

Step 2: Build a mock OpenAI-compatible server

The foundation of testing llm cli mock gpt-5 python is a faithful HTTP mock. Stand up a FastAPI app that implements POST /v1/chat/completions and echoes a canned response. This catches contract drift better than patching the SDK with unittest.mock.

# mock_server.py
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()
LAST_REQUEST = {}

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    payload = await request.json()
    LAST_REQUEST.clear()
    LAST_REQUEST.update(payload)
    assert payload["model"].startswith("gpt-5"), "unexpected model"
    return {
        "id": "chatcmpl-mock",
        "object": "chat.completion",
        "model": payload["model"],
        "choices": [
            {
                "index": 0,
                "message": {"role": "assistant", "content": "MOCK_RESPONSE"},
                "finish_reason": "stop",
            }
        ],
        "usage": {"prompt_tokens": 5, "completion_tokens": 2, "total_tokens": 7},
    }

@app.post("/v1/chat/completions/stream")
async def chat_completions_stream(request: Request):
    async def gen():
        yield b'data: {"choices":[{"delta":{"content":"MOCK"}}]}\n\n'
        yield b'data: {"choices":[{"delta":{"content":"_STREAM"}}]}\n\n'
        yield b"data: [DONE]\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream")

Asserting on model prevents a misconfigured test from silently hitting a different model family. Recording LAST_REQUEST lets tests assert the CLI sent the right messages.

Step 3: Redirect the CLI to the mock

The CLI already honors OPENAI_BASE_URL. In tests, set it to your local server before invoking the CLI. If you call main() directly, monkeypatch sys.argv and capsys to capture output.

# test_cli.py (snippet)
import sys
import pytest
from cli import main

def test_cli_with_env(monkeypatch, live_mock, capsys):
    monkeypatch.setenv("OPENAI_BASE_URL", live_mock)
    monkeypatch.setenv("OPENAI_API_KEY", "x")
    monkeypatch.setattr(sys, "argv", ["cli", "hello"])
    main()
    assert "MOCK_RESPONSE" in capsys.readouterr().out

Do not hardcode the mock URL in the CLI. Environment-based configuration keeps production and test code identical.

Step 4: Launch the mock in a pytest fixture

Use uvicorn in a background thread. This gives you a real TCP socket, so the OpenAI client behaves exactly as in production, including connection pooling and TLS-less localhost.

# test_cli.py (fixture)
import threading
import uvicorn
import time
import requests
import pytest

@pytest.fixture(scope="module")
def live_mock():
    config = uvicorn.Config(
        "mock_server:app", host="127.0.0.1", port=8099, log_level="error"
    )
    server = uvicorn.Server(config)
    t = threading.Thread(target=server.run, daemon=True)
    t.start()
    for _ in range(50):
        try:
            requests.post("http://127.0.0.1:8099/v1/chat/completions",
                          json={"model": "gpt-5", "messages": []}, timeout=0.1)
        except Exception:
            time.sleep(0.1)
        else:
            break
    yield "http://127.0.0.1:8099/v1"
    server.should_exit = True

The readiness loop prevents race conditions where the test runs before uvicorn finishes binding. Scope the fixture to module so you pay the startup cost once per test file.

Step 5: Mock streaming and error paths

Most CLIs stream. Point the CLI at the streaming endpoint by adding a route or branching on stream in the mock. Test that concatenated output equals MOCK_STREAM.

def test_cli_stream(monkeypatch, live_mock, capsys):
    monkeypatch.setenv("OPENAI_BASE_URL", live_mock + "/stream")
    monkeypatch.setenv("OPENAI_API_KEY", "x")
    monkeypatch.setattr(sys, "argv", ["cli", "hi", "--stream"])
    main()
    assert "MOCK_STREAM" in capsys.readouterr().out

Also mock a 429 to verify your retry/backoff. Add a route that raises HTTPException(status_code=429). Assert the CLI exits non-zero or retries per your policy. If you have no policy, write one—silent failure on rate limits is how you ship a broken tool.

Step 6: Verify success and swap to a real gateway

Verification is two-fold: automated and manual. Automated: pytest -q shows passed tests with the mock. Manual: run the CLI against the mock with OPENAI_BASE_URL=http://127.0.0.1:8099/v1 python cli.py "test" and confirm you see MOCK_RESPONSE instantly, with no outbound network calls except to localhost.

When you later point the same CLI at a real inference gateway, the OpenAI-compatible contract means no code changes. For instance, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is degraded, so the mock you built today validates the exact request shape you’ll send to production.

Step 7: Wire into CI

Add a GitHub Actions step that installs dependencies and runs the suite. No secrets needed because the mock never calls out.

# .github/workflows/test.yml
name: test
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install openai fastapi uvicorn pytest requests httpx
      - run: pytest -q

Keep the mock server in the repo. It doubles as a local dev stub when the real API is rate-limited or you are on a plane.

Step 8: Avoid common pitfalls

Don’t use unittest.mock.patch on the OpenAI client class. It hides changes in the JSON schema—like when GPT-5 adds a system_fingerprint field—and your tests stay green while prod breaks. Mock the HTTP layer.

Also, assert on the request the mock received. In the fixture teardown or a separate test, check LAST_REQUEST:

def test_cli_sent_correct_prompt(live_mock):
    # trigger a call via subprocess or main()
    import subprocess, os
    env = os.environ.copy()
    env["OPENAI_BASE_URL"] = live_mock
    env["OPENAI_API_KEY"] = "x"
    subprocess.run(["python", "cli.py", "verify-me"], env=env, check=True)
    from mock_server import LAST_REQUEST
    assert LAST_REQUEST["messages"][0]["content"] == "verify-me"

That catches prompt formatting regressions that otherwise surface only in user complaints.

If your tool uses function calling, mock the tools echo as well. Return choices[0].message.tool_calls with a fixed id and function name. Test that your CLI dispatches to the right local function. This is where most CLI LLM integrations actually break.

The full loop is: build CLI with env-injected base URL, mock the HTTP contract, test offline with pytest, verify manually, then ship against a real gateway. You now have a deterministic harness for testing llm cli mock gpt-5 python that survives model upgrades, SDK revisions, and provider outages.