A local-first workflow for building LLM apps

A local-first llm app workflow keeps your inner loop fast and your data on your machine. By running models locally during development and mocking remote APIs for tests, you can build agent logic, prompt chains, and eval harnesses without a cloud bill or compliance review. This guide lays out an ordered path from zero to a swappable architecture that survives contact with production.

1. Stand up a local model runtime

Ollama remains the lowest-friction way to run quantized LLMs on a laptop. Install it, pull a 7B–13B model, and expose the native API.

brew install ollama
ollama pull llama3.1:8b
ollama serve

The server listens on http://localhost:11434. Test it with a curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Say hi in JSON."
}'

For repeatable builds, pin a Modelfile instead of relying on CLI flags:

FROM llama3.1:8b
PARAMETER num_ctx 4096
PARAMETER temperature 0.2

Build it with ollama create mydev -f Modelfile. Pitfall: default context is 2048 tokens. For RAG or agent traces, bump it via Modelfile or API param. Tradeoff: larger context eats RAM and slows generation on CPU-only boxes. A 8B model at 4-bit uses ~6GB VRAM; if you are on integrated graphics, expect 5–10 tokens/sec.

2. Wrap everything in an OpenAI-compatible client

The fastest way to keep a local-first llm app workflow portable is to speak the OpenAI chat protocol locally. Ollama ships an /v1/chat/completions endpoint that mirrors the shape closely enough.

from openai import OpenAI

local = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # ignored, but required by SDK
)

resp = local.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarize: The quick brown fox."}],
)
print(resp.choices[0].message.content)

If you later swap to a remote gateway, only the base_url and api_key change. This is the seam that protects you from vendor lock-in. Note that Ollama’s model field expects the tag you pulled; the OpenAI SDK will pass it through verbatim.

3. Mock the endpoint for unit tests

Real local models are slow and non-deterministic. For CI and unit tests, mock the HTTP layer. Below is a minimal pytest + respx pattern that returns a fixed completion.

import respx
import httpx
from myapp.llm import local_client

@respx.mock
def test_summary():
    route = respx.post("http://localhost:11434/v1/chat/completions").mock(
        return_value=httpx.Response(200, json={
            "id": "x",
            "object": "chat.completion",
            "choices": [{"index":0,"message": {"role":"assistant","content": "Fox."},"finish_reason":"stop"}]
        })
    )
    out = local_client.summarize("The quick brown fox")
    assert out == "Fox."
    assert route.called

Tradeoff: mocks drift from real model behavior. Mitigate by snapshotting a handful of real responses and replaying them in integration tests weekly. Keep a tests/fixtures/ollama_smoke.json with captured payloads so you can assert on token counts and stop reasons, not just text.

4. Define a thin provider abstraction

Don’t scatter OpenAI() calls across modules. Create one interface that your app uses, so the local-first llm app workflow and cloud path share code.

from abc import ABC, abstractmethod
from openai import OpenAI

class LLMProvider(ABC):
    @abstractmethod
    def complete(self, messages: list, **kw) -> str: ...

class OllamaProvider(LLMProvider):
    def __init__(self, base_url: str, model: str):
        self.client = OpenAI(base_url=base_url, api_key="x")
        self.model = model
    def complete(self, messages, **kw):
        r = self.client.chat.completions.create(model=self.model, messages=messages, **kw)
        return r.choices[0].message.content

class FakeProvider(LLMProvider):
    def complete(self, messages, **kw):
        return "stub"

Now your business logic depends on LLMProvider, not on a network endpoint. In tests you inject a FakeProvider.

Avoid hidden global state

Don’t cache the client in a module-level singleton unless you also expose a reset hook. Tests will bleed state. Use dependency injection or a factory bound to your app context.

5. Keep prompts in versioned files

Prompt engineering is code. Store templates as .jinja or .txt in a prompts/ dir, load them at runtime, and diff changes in git.

from pathlib import Path
from jinja2 import Template

def load_prompt(name: str, **ctx) -> str:
    tpl = Template(Path(f"prompts/{name}.jinja").read_text())
    return tpl.render(**ctx)

This makes the local-first llm app workflow reviewable. A teammate can propose a prompt tweak via PR without spinning up the model. For multi-step agents, keep each step’s prompt separate; never concatenate strings inline.

6. Test streaming and tool calls explicitly

Local runtimes handle streaming differently than hosted ones. Ollama streams via SSE on the same chat endpoint; verify your client consumes it.

stream = local.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role":"user","content":"Count to 3"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Tool calling is where local models often diverge. Llama 3.1 supports JSON mode but not all function-calling schemas. If your app relies on parallel tool calls, test against the actual local model early; don’t assume parity with GPT-4o. Write a contract test that sends a tools payload and asserts the response shape, not just the content.

7. Run local evals before touching the cloud

Build a small eval set: 20–50 inputs with expected substrings or rubric scores. Run them against Ollama overnight.

import json, asyncio
from pathlib import Path
from myapp.llm import OllamaProvider

def eval_set(path: str):
    prov = OllamaProvider("http://localhost:11434/v1", "llama3.1:8b")
    cases = json.loads(Path(path).read_text())
    fails = 0
    for c in cases:
        out = prov.complete([{"role":"user","content":c["input"]}])
        if c["expect"] not in out:
            fails += 1
            print(f"fail: {c['input']} -> {out}")
    print(f"{len(cases)-fails}/{len(cases)} passed")

If the local model can’t clear 80% on your smoke eval, a bigger model or different quantization is needed before spending remote tokens. Use the same eval harness in CI with the mocked provider to catch regressions in prompt templates.

8. Promote to production via the same interface

When the app is proven, change the base_url to a hosted gateway. For example, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and automatically falls back when a provider is rate-limited, so the OllamaProvider swap is a one-line config change.

prod = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key=os.environ["N4N_KEY"],
)

Keep the local path as a dev profile. This preserves the local-first llm app workflow for every new feature: build locally, mock in CI, eval on Ollama, ship via gateway. The gateway also forwards provider cache-control hints, which your local server ignores—so validate caching on a staging key before launch.

9. Common pitfalls and tradeoffs

Latency blindness. Local CPU inference might take 30s for a 500-token reply. Your UI timeout of 5s will fail in prod. Always test with a representative concurrency model or stub a slower mock.

Tokenization mismatch. Ollama uses model-native tokenizers. Prompt lengths measured locally can exceed remote context windows by 10–20%. Log token counts from the API response, not from len(text.split()).

Cache-control hints ignored. Local servers don’t honor cache_control breakpoints. If you rely on provider prompt caching to cut cost, your local tests won’t exercise that path. Use a remote staging account for cache validation.

Model drift. A quantized 8B model behaves differently from a 70B or a frontier API. Treat local passes as necessary but not sufficient; run a final eval on the production model family.

Silent schema changes. Ollama’s /v1 response sometimes omits system_fingerprint. Code that branches on that field will throw only in prod.

10. Minimal repo layout

prompts/
  summarize.jinja
  agent_step1.jinja
tests/
  test_mock.py
  test_integration.py
  fixtures/ollama_smoke.json
myapp/
  llm.py        # provider abstraction
  cli.py        # dev runner
eval/
  smoke.json

Stick to this and the local-first llm app workflow stays boring—in the good way. You iterate fast, test deterministically, and promote without rewrites. The moment a feature needs a model you can’t run locally, you flip a config flag, not an architecture.