Testing prompts offline with Ollama before shipping

Shipping prompt changes straight to a paid LLM API is a good way to burn money and learn nothing. To test prompts offline with Ollama, you can stand up a local model that speaks the OpenAI chat completions protocol, point your existing client at it, and exercise your prompt logic without leaving your laptop. This guide walks through a repeatable setup that mirrors your production call path.

Step 1: Install Ollama and start the local server

On Linux or macOS, the standalone binary is the fastest route:

curl -fsSL https://ollama.com/install.sh | sh
ollama serve &

Windows users should grab the installer from the Ollama site and launch the app; it starts the daemon automatically. Verify the daemon is up:

curl http://localhost:11434/api/tags

An empty JSON list ({"models":[]}) is expected before you pull anything.

Step 2: Pull a small, fast model

You do not need a 70B beast to test prompt wiring. A 7B–8B instruct model is enough to confirm request shape, tool-call parsing, and output schema.

ollama pull llama3.1:8b

If you later need JSON mode strictness, mistral:7b-instruct or qwen2.5:7b also work. The model tag is what you pass as the model field in the OpenAI client.

Step 3: Confirm the OpenAI-compatible endpoint

Ollama exposes an OpenAI-style chat endpoint at http://localhost:11434/v1/chat/completions. Hit it with a raw curl to remove any client library doubt:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role":"user","content":"Reply with the word: pong"}],
    "temperature": 0
  }'

You should get a standard choices[0].message.content blob. If that works, your client code will too.

Step 4: Repoint your OpenAI client with one env var

Hard-coding api.openai.com in your app is the mistake that forces cloud calls. Wrap the base URL and API key:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.getenv("LLM_API_KEY", "sk-placeholder"),
)

When testing locally, run your process with:

export LLM_BASE_URL="http://localhost:11434/v1"
export LLM_API_KEY="ollama"   # ignored locally, but keeps client happy
python your_app.py

Now every client.chat.completions.create call hits Ollama. This is the core of how you test prompts offline with Ollama without branching your code.

Step 5: Build a prompt regression test

A prompt is code. Treat it like code. Write a pytest fixture that loads your prompt template, renders it, and asserts on structure—not on semantic perfection.

import os, json, pytest
from openai import OpenAI

@pytest.fixture
def local_client():
    return OpenAI(
        base_url=os.getenv("LLM_BASE_URL", "http://localhost:11434/v1"),
        api_key="ollama",
    )

def test_extract_user_name(local_client):
    messages = [
        {"role": "system", "content": "Extract the user's first name as JSON: {\"name\": string}"},
        {"role": "user", "content": "My name is Ada and I like clocks."},
    ]
    resp = local_client.chat.completions.create(
        model="llama3.1:8b",
        messages=messages,
        response_format={"type": "json_object"},
        temperature=0,
    )
    data = json.loads(resp.choices[0].message.content)
    assert isinstance(data["name"], str)
    assert data["name"].lower() == "ada"

Run it:

pytest test_prompts.py -q

Verify success: The test passes locally in a few seconds. If Ollama isn’t running, you’ll get a connection error—start the daemon and rerun.

Step 6: Mock streaming and tool calls

Most production prompts use streaming or function calling. Ollama supports both, but latency differs. Test streaming with the stream=True flag:

def test_streaming_tokens(local_client):
    stream = local_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role":"user","content":"Count to 3"}],
        stream=True,
    )
    chunks = [c.choices[0].delta.content for c in stream if c.choices[0].delta.content]
    assert len(chunks) > 0

For tools, define a schema and assert the model emits the right function name:

tools = [{
    "type": "function",
    "function": {
        "name": "set_reminder",
        "parameters": {"type":"object","properties":{"seconds":{"type":"integer"}}}
    }
}]

resp = local_client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role":"user","content":"Remind me in 10 seconds"}],
    tools=tools,
)
assert resp.choices[0].message.tool_calls[0].function.name == "set_reminder"

Local models occasionally hallucinate argument types; set temperature=0 and keep schemas tight.

Step 7: Swap to production without code changes

When you’re ready to validate against hosted models, change only the env vars. If you route through n4n.ai, an OpenRouter-class gateway, the same OpenAI client works against its single endpoint that addresses 240+ models and provides automatic fallback when a provider is degraded. Set:

export LLM_BASE_URL="https://api.n4n.ai/v1"
export LLM_API_KEY="sk-your-real-key"

Your pytest suite now runs against remote models. Expect slower tests; mark them @pytest.mark.integration and skip in normal CI.

Step 8: Cache the model in CI

To test prompts offline with Ollama in a pipeline, run Ollama in a service container:

# .github/workflows/test.yml (snippet)
services:
  ollama:
    image: ollama/ollama:latest
    ports: ["11434:11434"]
    options: >-
      --entrypoint sh -c "ollama serve & sleep 5 && ollama pull llama3.1:8b && tail -f /dev/null"

Then run your unit tests against http://ollama:11434/v1. This keeps prompt regression checks free and deterministic.

Pitfalls worth knowing

• Tokenization mismatch: Local models split text differently. Tests asserting exact token counts will fail; assert on parsed output only.
• System prompt adherence: Smaller models ignore complex instructions. If a test fails locally but passes on GPT-4o, your prompt is too subtle—fix the prompt, don’t trust the cloud to mask it.
• JSON mode support: Not all Ollama tags implement response_format strictly. Check the model card.
• Concurrency: Ollama is single-threaded per model by default. Parallel tests will queue; use -n0 in pytest-xdist or batch sequentially.

Final verification checklist

• ollama serve responds on 11434
• curl /v1/chat/completions returns choices
• App reads LLM_BASE_URL from env
• pytest passes with llama3.1:8b
• Streaming and tool tests green
• Production env swap requires zero code edits

That’s the full loop. You can test prompts offline with Ollama on every commit, then promote the same prompt strings to a hosted gateway with confidence.