Setting up a local LLM dev environment without API costs

Running a local llm dev environment no api cost is the fastest way to iterate on LLM app logic without burning dollars on every typo. This tutorial wires Ollama, a lightweight OpenAI-compatible endpoint, and a mock server so your existing SDK calls work unchanged against models on your machine.

Prerequisites

• Python 3.10+ with pip
• Ollama installed (macOS, Linux, or Windows)
• curl for quick checks
• fastapi and uvicorn for the mock server (installed below)
• Familiarity with the OpenAI Python client

If you haven’t installed Ollama yet, do that first. Everything else we’ll install as we go.

Step 1: Install Ollama and pull a model

Ollama ships a single binary that runs quantized models locally. On macOS or Linux:

curl -fsSL https://ollama.com/install.sh | sh

Windows users should grab the installer from the Ollama site. After install, pull a small general-purpose model:

ollama pull llama3.1:8b

Expected output ends with:

pulling manifest
success

Verify the daemon is up:

curl http://localhost:11434/api/tags

You’ll see JSON listing llama3.1:8b. That confirms your local llm dev environment no api cost is live.

Step 2: Use Ollama’s OpenAI-compatible endpoint

Ollama exposes an OpenAI-compatible chat API at http://localhost:11434/v1. No extra proxy needed. Test it directly:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Say hi in one word."}],
    "max_tokens": 10
  }'

Expected shape (truncated):

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "llama3.1:8b",
  "choices": [
    { "message": { "role": "assistant", "content": "Hi." } }
  ]
}

This is a real completion, served from your CPU/GPU, with zero per-call fees.

Step 3: Point the OpenAI client at local Ollama

In your Python code, set base_url and a dummy api_key. The client library doesn’t care that the backend is local.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # ignored by Ollama, required by SDK
)

resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "What is 2+2?"}],
)
print(resp.choices[0].message.content)

Run it:

python app.py

Output:

4

You now have a working local llm dev environment no api cost that uses the exact same interface as production.

Step 4: Build a mock LLM server for deterministic tests

Local models are free but slow and non-deterministic. For unit tests you want a stub that returns fixed JSON. Here’s a minimal FastAPI mock:

# mock_server.py
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
import uvicorn

app = FastAPI()
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"])

@app.post("/v1/chat/completions")
async def chat(payload: dict):
    prompt = payload["messages"][-1]["content"]
    return {
        "id": "mock-1",
        "object": "chat.completion",
        "model": payload.get("model", "mock"),
        "choices": [
            {
                "index": 0,
                "message": {"role": "assistant", "content": f"ACK: {prompt}"},
                "finish_reason": "stop",
            }
        ],
        "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
    }

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=8000)

Start it:

pip install fastapi uvicorn
python mock_server.py

Checkpoint:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mock","messages":[{"role":"user","content":"test"}]}'

Returns:

{"id":"mock-1","object":"chat.completion","model":"mock","choices":[{"index":0,"message":{"role":"assistant","content":"ACK: test"},"finish_reason":"stop"}]}

Step 5: Switch backends via environment

Hard-coding URLs is a mistake. Read them from the environment so the same code runs against Ollama, the mock, or a cloud gateway.

# config.py
import os
from openai import OpenAI

LLM_BASE_URL = os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1")
LLM_API_KEY = os.environ.get("LLM_API_KEY", "ollama")
LLM_MODEL = os.environ.get("LLM_MODEL", "llama3.1:8b")

client = OpenAI(base_url=LLM_BASE_URL, api_key=LLM_API_KEY)

def ask(prompt: str) -> str:
    resp = client.chat.completions.create(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

.env for local dev:

LLM_BASE_URL=http://localhost:11434/v1
LLM_API_KEY=ollama
LLM_MODEL=llama3.1:8b

.env.test for mock:

LLM_BASE_URL=http://127.0.0.1:8000/v1
LLM_API_KEY=fake
LLM_MODEL=mock

Load with python-dotenv or your shell. This keeps your local llm dev environment no api cost flexible.

Step 6: Write a small app and test both backends

A tiny CLI that exercises the switch:

# cli.py
import sys
from config import ask

if __name__ == "__main__":
    prompt = sys.argv[1] if len(sys.argv) > 1 else "Hello"
    print(ask(prompt))

Run against Ollama:

python cli.py "Capital of France?"

Sample output:

Paris.

Run against mock:

LLM_BASE_URL=http://127.0.0.1:8000/v1 LLM_API_KEY=fake LLM_MODEL=mock python cli.py "ping"

Output:

ACK: ping

Your test suite can boot the mock on a port and set LLM_BASE_URL in fixtures. No network, no cost, millisecond responses.

Step 7: Add a Makefile for repeatability

Engineers hate memorizing commands. Wrap it:

# Makefile
install:
	pip install openai fastapi uvicorn python-dotenv

mock:
	python mock_server.py

local:
	python cli.py "What is 2+2?"

test:
	LLM_BASE_URL=http://127.0.0.1:8000/v1 LLM_API_KEY=fake LLM_MODEL=mock python cli.py "test"

Now make test runs deterministic checks; make local hits the real model.

Handling streaming and tool calls

Ollama’s OpenAI shim supports streaming. Adjust ask to yield tokens:

def ask_stream(prompt: str):
    stream = client.chat.completions.create(
        model=LLM_MODEL,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

The mock can also stream if you return text/event-stream, but for unit tests a non-streaming stub is usually enough.

When you need hosted models without the key juggling

A local llm dev environment no api cost covers iteration and tests, but sometimes you need to validate against a larger model before shipping. n4n.ai provides one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded, which lets you point the same OpenAI client at staging without rewriting code or managing multiple API keys.

Closing checklist

• Ollama running with a pulled model
• OpenAI client pointed at localhost:11434/v1
• FastAPI mock for fast tests
• Env-driven base URL and model name
• Makefile or CI step to swap backends

That’s a complete, free, reproducible setup. You can develop, test, and refactor LLM features without a single billed token.