Caching LLM API calls in CI to cut test costs

Running LLM-powered tests in continuous integration burns money because every test run hits a paid API. Caching LLM API calls in CI lets you replay deterministic responses from previous runs, turning a per-call charge into a free local read. This guide gives you an end-to-end pattern with pytest and GitHub Actions that you can adapt to any OpenAI-compatible endpoint.

Prerequisites

Python 3.10+ and pytest installed.
openai Python SDK (v1.x) or equivalent HTTP client.
A CI provider with job artifact caching (GitHub Actions used below).
Tests that call an LLM with fixed parameters.

Step 1: Isolate deterministic test calls

Identify which tests actually need the network. A call is cacheable only if the same request yields the same response within your tolerance window. Set temperature=0 and pass a seed when the provider allows it. Keep the model pinned to a dated snapshot.

Refactor business logic to accept a client object rather than constructing one internally. That makes injection of a cache wrapper trivial.

# app/classifier.py
import openai

def classify(text: str, client: openai.OpenAI) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini-2024-07-18",
        messages=[{"role": "user", "content": f"Classify sentiment: {text}"}],
        temperature=0,
    )
    return resp.choices[0].message.content

# tests/test_classifier.py
from app.classifier import classify

def test_positive():
    client = openai.OpenAI()  # replaced by fixture later
    assert "positive" in classify("I love this", client).lower()

Any test using randomized sampling, current events, or tool calls with side effects should stay live or be mocked with static fixtures.

Step 2: Choose a cache store that survives CI runs

Do not commit cached JSON to git. Use a directory restored by your CI cache mechanism. A flat key-value store on disk is enough for thousands of entries.

mkdir -p .llm_cache
echo ".llm_cache/" >> .gitignore

For larger teams, Redis works but adds infrastructure. File cache keeps the how-to simple and portable across runners.

Step 3: Wrap the client with a hash-based cache

Build a proxy that hashes the exact request payload. Include the endpoint path, model, messages, and all sampling params. On hit, return the stored dict. On miss, call the real API and write the response.

import hashlib, json, os, time, openai

CACHE_DIR = os.environ.get("LLM_CACHE_DIR", ".llm_cache")
MODE = os.environ.get("LLM_CACHE_MODE", "rw")  # rw, ro, write

class CachedClient:
    def __init__(self, base_url, api_key):
        self._client = openai.OpenAI(base_url=base_url, api_key=api_key)

    def _key(self, kwargs):
        # exclude stream from hash if you aggregate later
        k = {kk: vv for kk, vv in kwargs.items() if kk != "stream"}
        payload = json.dumps(k, sort_keys=True, default=str)
        return hashlib.sha256(payload.encode()).hexdigest()

    def chat(self):
        return self

    def completions(self):
        return self

    def create(self, **kwargs):
        key = self._key(kwargs)
        path = os.path.join(CACHE_DIR, key + ".json")
        if MODE != "write" and os.path.exists(path):
            with open(path) as f:
                return json.load(f)
        if MODE == "ro":
            raise AssertionError(f"LLM cache miss (read-only): {key}")
        resp = self._client.chat.completions.create(**kwargs)
        data = resp.model_dump() if hasattr(resp, "model_dump") else resp.dict()
        data["_meta"] = {"model": kwargs.get("model"), "cached_at": time.time()}
        os.makedirs(CACHE_DIR, exist_ok=True)
        with open(path, "w") as f:
            json.dump(data, f)
        return data

# Fixture to inject
import pytest

@pytest.fixture
def llm_client(monkeypatch):
    inst = CachedClient(base_url=os.environ["OPENAI_BASE_URL"], api_key="fake")
    monkeypatch.setattr(openai, "OpenAI", lambda *a, **k: inst)
    return inst

Now your existing test passes without modification because openai.OpenAI() returns the cached proxy. The hash covers every parameter, so a prompt edit forces a fresh call.

Reconstructing SDK objects

If your code calls methods on the response (e.g., resp.choices[0].message.content), the dict works with bracket access but not attribute access. Either use resp["choices"][0]["message"]["content"] in app code, or rebuild the pydantic object with openai.types.chat.ChatCompletion(**data). For tests, dict is fine.

Step 4: Wire cache restore/save into GitHub Actions

Add caching around the test job. Key on a hash of the test directory so prompt or assertion changes invalidate the store, while unrelated file changes keep it warm.

name: CI
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    env:
      LLM_CACHE_DIR: .llm_cache
      OPENAI_BASE_URL: https://api.example.com/v1
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Restore LLM cache
        uses: actions/cache@v4
        with:
          path: .llm_cache
          key: llm-${{ hashFiles('tests/**/*.py') }}
          restore-keys: llm-
      - run: pip install openai pytest
      - name: Populate or verify cache
        run: pytest
      - name: Save LLM cache
        if: always()
        uses: actions/cache@v4
        with:
          path: .llm_cache
          key: llm-${{ hashFiles('tests/**/*.py') }}

For GitLab CI, use cache: with key: files: tests/**/*.py and paths: [.llm_cache]. The principle is identical.

Step 5: Pin models and honor cache-control hints

Floating model aliases (gpt-4o-mini) change underneath you. Pin to a dated version. Cache entries should carry metadata to support eviction.

# eviction script (run periodically)
import os, time, json
for f in os.listdir(CACHE_DIR):
    if not f.endswith(".json"): continue
    with open(os.path.join(CACHE_DIR, f)) as fh:
        meta = json.load(fh).get("_meta", {})
    if time.time() - meta.get("cached_at", 0) > 604800:  # 7 days
        os.remove(os.path.join(CACHE_DIR, f))

If you route through n4n.ai, its OpenAI-compatible endpoint addresses 240+ models and forwards provider cache-control hints; parse those hints to set per-model TTLs instead of a blanket week. That keeps you aligned with upstream stability without manual tuning.

Step 6: Verify the cache works and costs drop

Verification has two parts: correctness and cost. First, ensure tests pass with a warm cache. Second, enforce that no network calls happen on repeated runs.

Run locally:

pytest                     # miss, populates .llm_cache
LLM_CACHE_MODE=ro pytest   # must pass with zero misses

In CI, add a second step that sets LLM_CACHE_MODE=ro after the initial populate job. If a test changes without cache invalidation, it fails loudly.

      - name: Test read-only (no cost)
        run: LLM_CACHE_MODE=ro pytest

Counting saved calls

Add a small metric to the wrapper: increment a counter on hit/miss and print at session end via pytest_sessionfinish.

# conftest.py
import pytest

def pytest_sessionfinish(session, exitstatus):
    hits = getattr(session.config, "llm_hits", 0)
    misses = getattr(session.config, "llm_misses", 0)
    print(f"\nLLM cache hits: {hits}, misses: {misses}")

A healthy suite shows hits > 0 and misses == 0 on every run after the first. Check your provider’s per-token usage metering to confirm zero completion tokens are billed on warm runs. That proves caching LLM API calls in CI is active.

Gotchas

Streaming: Aggregate the stream into a single string before hashing and storing.
Parallel matrices: Suffix CACHE_DIR with the matrix value to avoid concurrent writes clobbering files.
Prompt injection in tests: Cache files contain your prompts; restrict CI log access.
Model deprecation: When a pinned model is retired, the live call will 404; your read-only mode will then error, signaling you to update the pin and refresh cache.

Caching LLM API calls in CI is straightforward engineering, not an exotic optimization. Wrap the client, persist the responses, and switch to read-only in CI to guarantee you never pay twice for the same token.