n4nAI
Open console →
Developer toolsHow-toCI/CD Pipelines for LLM Apps

How to keep CI fast when every test hits an LLM API

Practical steps for keeping CI fast with LLM API test calls: mock the client, replay fixtures, use small models, cache layers, and enforce budgets.

n4n Team3 min read725 words

Audio narration

Coming soon — every post will get a voice note here.

Every team that wires tests to a live model eventually watches CI rot into a 40-minute slog of rate-limit errors and nondeterministic failures. Keeping CI fast with LLM API test calls means treating the model as a slow, expensive I/O boundary that you stub, record, or shrink—not a service you trust in every run.

Step 1: Classify tests by real model dependency

Before changing tooling, map your suite. A prompt-builder unit test has no business hitting an API. An integration test that validates JSON parsing against a real response does. A weekly eval that checks output quality needs a live model but not in every commit.

Keeping CI fast with LLM API test calls starts with refusing to pay for model latency where you aren’t testing the model. Split your pytest targets into a clear tree:

tests/
  unit/          # no network, fake client
  integration/   # recorded or cached responses
  eval/          # live small model, nightly only

Tag them so you can run fast subsets in pre-commit:

# tests/integration/test_parse.py
import pytest

@pytest.mark.integration
def test_extract_json(llm_replay):
    out = llm_replay.complete("Return JSON: {name: 'x'}")
    assert '"name"' in out

Run only unit in the primary CI stage: pytest tests/unit -q. The goal is to make the commit gate independent of third-party uptime.

Step 2: Mock the client at the boundary

Define a narrow interface and inject a fake in tests. This removes all network calls from logic tests and makes them deterministic.

# llm/client.py
from abc import ABC, abstractmethod

class LLMClient(ABC):
    @abstractmethod
    def complete(self, prompt: str) -> str: ...

class FakeLLMClient(LLMClient):
    def complete(self, prompt: str) -> str:
        # Canned text shaped like real output
        return '{"name": "x"}'
# conftest.py
import pytest
from llm.client import FakeLLMClient

@pytest.fixture
def llm_client():
    return FakeLLMClient()

Any test using llm_client runs in microseconds. This is the single highest-leverage change for keeping CI fast with LLM API test calls, because it deletes the variable you can’t control.

Step 3: Record and replay real responses for integration tests

For tests that must exercise the real request shape, call the API once locally, save the response, and replay it in CI. Use a hash of the normalized prompt as the key.

# tests/integration/replay.py
import json, hashlib, os
from llm.client import LLMClient

class ReplayClient(LLMClient):
    def __init__(self, mode="replay", record_dir="tests/fixtures"):
        self.mode = mode
        self.record_dir = record_dir

    def complete(self, prompt: str) -> str:
        key = hashlib.sha256(prompt.strip().encode()).hexdigest()
        path = os.path.join(self.record_dir, f"{key}.json")
        if self.mode == "replay":
            with open(path) as f:
                return json.load(f)["response"]
        # record mode: hit real API, persist, return
        real = real_client.complete(prompt)
        with open(path, "w") as f:
            json.dump({"response": real}, f)
        return real

Generate fixtures with mode="record" on a developer machine, then commit the JSON. CI never leaves the repo. Normalize prompts (strip whitespace, sort JSON keys) before hashing to avoid cache misses from insignificant changes.

Verify replay works

Block network and run integration offline:

pytest tests/integration --disable-socket

They should pass using only fixtures.

Step 4: Use a tiny model for the few live smoke tests

Some paths—auth headers, streaming parsing, tool-call shape—need a live call. Use the smallest, cheapest model your provider offers and keep the prompt trivial.

# CI env
export LLM_MODEL=text-babbage-002
export LLM_BASE_URL=https://api.openai.com/v1
import os
from openai import OpenAI

def live_client():
    return OpenAI(api_key=os.environ["LLM_API_KEY"]).chat.completions

def test_smoke():
    resp = live_client().create(
        model=os.environ["LLM_MODEL"],
        messages=[{"role": "user", "content": "say OK"}],
        timeout=5,
    )
    assert resp.choices[0].message.content.strip().upper() == "OK"

Run these in a separate stage with a concurrency limit of 1. They are a canary, not a test of intelligence.

Step 5: Put a caching proxy in front of the API

If you must call live models in CI, cache identical prompts. An OpenAI-compatible gateway such as n4n.ai addresses 240+ models behind one endpoint, honors client cache-control hints, and provides automatic fallback when a provider is degraded—wire it as your base URL and repeated prompts cost zero extra latency after the first hit. It also forwards routing directives, so you can pin a specific provider per test type.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "https://api.n4n.ai/v1"),
    api_key=os.getenv("LLM_API_KEY"),
    timeout=5,
    max_retries=1,
)

def cached_complete(prompt: str) -> str:
    resp = client.chat.completions.create(
        model=os.getenv("LLM_MODEL", "gpt-3.5-turbo"),
        messages=[{"role": "user", "content": prompt}],
        extra_headers={"cache-control": "max-age=86400"},
    )
    return resp.choices[0].message.content

This step alone can cut repeated eval suites from minutes to seconds, and the fallback hides transient provider 429s.

Step 6: Enforce timeout and retry budgets

LLM endpoints fail weirdly. Set aggressive timeouts and disable SDK auto-retry storms.

client = OpenAI(timeout=5, max_retries=1)

In pytest, fail the test if it exceeds a budget:

import time
def test_within_budget():
    start = time.time()
    cached_complete("hi")
    assert time.time() - start < 5

A test that hangs on a model is a CI resource leak; kill it fast.

Step 7: Limit parallelism to avoid rate limits

pytest-xdist helps, but unbounded concurrency triggers 429s that look like bugs. Cap workers and separate stages in your pipeline config.

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Unit
        run: pytest tests/unit -n 2
      - name: Integration (replay)
        run: pytest tests/integration --disable-socket
      - name: Live smoke
        if: github.ref == 'refs/heads/main'
        run: pytest tests/eval -n 1

If you use the gateway from Step 5, its fallback masks transient provider limits, but you still pay latency, so keep live concurrency low.

Step 8: Meter token usage to catch regressions

Log usage on every live call. A sudden jump in prompt tokens means a test fixture or prompt template ballooned.

resp = client.chat.completions.create(...)
print(f"tokens={resp.usage.total_tokens}")

If you route through a gateway with per-token usage metering, export those metrics to your CI dashboard. Keeping CI fast with LLM API test calls also means keeping it cheap—unbounded eval loops will quietly drain budget.

Step 9: Verify success

After applying Steps 1–8, measure the primary gate:

pytest tests/unit tests/integration --durations=10

Success criteria:

  • Unit + integration stage completes under 2 minutes with no network egress.
  • Live smoke stage runs <30s and only on main branch or nightly.
  • Token spend in CI drops >90% versus the unmodified suite (compare gateway metering reports).

If those hold, you’ve solved the problem. If integration tests still hit the wire, revisit Step 3’s replay hash—prompt normalization prevents cache misses. Treat the model like a database: seed it, fake it, or cache it. Never let it dictate your commit cycle.

Tagsci-cdperformancetestingllm-apps

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsA CI/CD pipeline for LLM apps, step by stepHands-on guide to building a CI/CD pipeline for LLM apps with GitHub Actions: validate prompts, run eval tests, gate deployments, and ship reliably.Developer toolsA GitHub Actions workflow for n4n-powered appsBuild a practical CI pipeline that tests and smoke-checks LLM apps using n4n.ai's OpenAI-compatible gateway, with GitHub Actions and live fallback validation.Developer toolsBuilding a CI/CD pipeline for a RAG applicationA hands-on tutorial to build a CI/CD pipeline for RAG applications using GitHub Actions, covering indexing, retrieval tests, and automated LLM evals.Developer toolsCaching LLM API calls in CI to cut test costsA practical guide to caching LLM API calls in CI pipelines using pytest and GitHub Actions, reducing test spend by replaying deterministic model responses.Developer toolsCI/CD for LLM apps vs traditional software: what changesHead-to-head comparison of CI/CD for LLM apps vs traditional software across capabilities, cost, latency, ergonomics, ecosystem, and limits.Developer toolsCost budgets in CI: capping token spend on every test runLearn how to implement capping token spend in CI test runs with a step-by-step guardrail using env vars, usage tracking, and CI config for LLM apps.

All ci/cd pipelines for llm apps posts →