Running Mistral and Llama models locally for offline dev

When you need to run Mistral Llama locally offline, the fastest path is Ollama plus a thin OpenAI-compatible shim. This tutorial builds a reproducible local setup that serves Mistral and Llama models over a localhost endpoint, so you can develop and test LLM features without burning cloud quota or leaking data.

Prerequisites

• A machine with 16 GB RAM minimum. Mistral 7B and Llama 3 8B in Q4 quantization fit in ~4–6 GB VRAM/RAM.
• Ollama installed. On Linux/macOS:

  curl -fsSL https://ollama.com/install.sh | sh

• Python 3.11+ and the OpenAI client library:

  pip install openai

• No network access required after model pulls. That is the whole point of the offline workflow.

Pull the models

Ollama distributes models as tagged images. Pull the two we care about:

ollama pull mistral
ollama pull llama3

mistral resolves to the latest 7B instruct build. llama3 pulls Meta’s 8B instruct. If you need a smaller footprint, append :7b-instruct-q4_0 or similar; run ollama list to see what landed.

Expected output ends with:

pulling manifest
success

Verify raw inference

Run a one-shot prompt to confirm the runtime works:

ollama run mistral "Return a JSON object with key 'ok' and value true."

You should see a well-formed JSON string. If you get a hang, check ollama serve is running (ps aux | grep ollama). The daemon binds to 127.0.0.1:11434 by default.

Use the OpenAI-compatible endpoint

Ollama exposes an OpenAI-compatible chat endpoint at /v1/chat/completions. Point the standard client at it:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # ignored locally, required by client
)

resp = client.chat.completions.create(
    model="mistral",
    messages=[{"role": "user", "content": "Ping"}],
    temperature=0,
)
print(resp.choices[0].message.content)

Run it. Output is a non-empty string like Pong or a short acknowledgement. The response object mirrors the OpenAI schema, so your production code paths stay unchanged.

Streaming works the same way

stream = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Count to 3."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

This prints tokens incrementally. Use streaming in local dev to catch latency regressions before they hit prod.

Swap local and remote with one env var

Hard-coding localhost in tests is a smell. Wrap the client in a factory:

import os
from openai import OpenAI

def get_client():
    base = os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1")
    key = os.environ.get("LLM_API_KEY", "ollama")
    return OpenAI(base_url=base, api_key=key)

client = get_client()

Set LLM_BASE_URL=https://api.n4n.ai/v1 and a real key to target a hosted gateway; the call site does not change because both speak the same OpenAI-compatible contract. That symmetry is what makes local offline dev a drop-in substitute rather than a fork.

Mock the API for unit tests

Sometimes you do not want to load 4 GB of weights just to test a parser. Stand up a 20-line mock with FastAPI:

from fastapi import FastAPI
from fastapi.testclient import TestClient

app = FastAPI()

@app.post("/v1/chat/completions")
async def fake_completion(payload: dict):
    return {
        "choices": [
            {"message": {"role": "assistant", "content": "{\"ok\": true}"}}
        ]
    }

client = TestClient(app)

Point LLM_BASE_URL at this mock in CI and your parsing logic runs in milliseconds. Keep the real Ollama run for integration tests only.

Keep models loaded between calls

Ollama unloads models after a short idle window. In a tight dev loop that causes a multi-second reload per request. Pin it:

ollama run mistral --keepalive 1h

Or set it per-request via the keep_alive field if you use the native API. For the OpenAI shim, prefix the model name is not supported; use the CLI flag or the /api/generate native endpoint when you need fine control.

Offline caveats

• Tokenizers: Ollama bundles tokenizers; no network needed at runtime. Do not call tiktoken for these models—use the model’s own counts from the response usage field if present, or accept approximate local counts.
• Concurrency: The default daemon handles one model at a time per GPU. Running Mistral and Llama simultaneously on a 16 GB box will swap to CPU and crawl. Load one, test, swap.
• Version drift: ollama pull updates silently. Pin with explicit tags in a Modelfile or a requirements.txt-style lock note so your offline dev matches what you tested.

A minimal dev script

Put it together in local_llm.py:

import os
from openai import OpenAI

def client():
    return OpenAI(
        base_url=os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1"),
        api_key=os.environ.get("LLM_API_KEY", "ollama"),
    )

if __name__ == "__main__":
    c = client()
    model = os.environ.get("LLM_MODEL", "mistral")
    out = c.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Say hi in JSON."}],
    )
    print(out.choices[0].message.content)

Run with LLM_MODEL=llama3 python local_llm.py. You now have a single command to run Mistral Llama locally offline against either model, swappable to any OpenAI-compatible gateway.

What we skipped

We did not cover fine-tuning, RAG pipelines, or multi-GPU serving. Those are unrelated to getting a credible local mock running today. For that, the setup above is enough to freeze your external dependencies and keep shipping while the wifi is off.