Running Llama 3 locally with Ollama for development

If you need to run llama 3 locally ollama is the fastest path to a working OpenAI-compatible endpoint on your laptop. This tutorial walks through installing Ollama, pulling the Llama 3 weights, and wiring the local model into your Python app so you can develop and test without burning cloud credits or leaking data.

Prerequisites

• Linux, macOS, or Windows Subsystem for Linux (WSL2).
• curl and python3.10+ with pip.
• At least 8 GB RAM (16 GB recommended for the 8B model); a CUDA or Metal GPU helps but isn’t required.
• No API keys or accounts.

Install Ollama and pull Llama 3

On macOS or Linux, install the daemon with the official script:

curl -fsSL https://ollama.com/install.sh | sh

On macOS it launches on install. On Linux, start it in the background:

ollama serve &

Pull the model. The llama3 tag defaults to the 8B parameter build:

ollama pull llama3:8b

Expected output:

pulling manifest
pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB
verifying sha256 digest
writing manifest
removing any unused layers
success

If you have 40+ GB VRAM, ollama pull llama3:70b works the same way.

Verify the local server

Ollama listens on 11434. Confirm the API responds:

curl http://localhost:11434/api/tags

You should get JSON similar to:

{"models":[{"name":"llama3:8b","size":4823421920,"digest":"6a0746a1ec1a..."}]}

Connection refused means the daemon died—check ollama serve logs.

Talk to Llama 3 from Python

Ollama mirrors the OpenAI chat completions schema at /v1. Install the client:

pip install openai

Then point it at the local endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="not-needed",
)

resp = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Explain what an LLM gateway does in one sentence."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Sample output:

An LLM gateway routes requests to multiple model providers, handling auth, fallbacks, and usage metering.

That is the core: you can now run llama 3 locally ollama and call it exactly like you would call a hosted model.

Streaming responses

For chat UIs you want tokens as they generate. Flip stream=True:

stream = client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "List three caching strategies for LLM apps."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Local generation on CPU runs at roughly 5–15 tokens/sec for 8B; a mid-range GPU pushes that to 40+.

Extend context with a Modelfile

Llama 3 8B supports 8k tokens, but Ollama defaults to 2k context. Create a custom build:

cat > Modelfile <<'EOF'
FROM llama3:8b
PARAMETER num_ctx 8192
EOF
ollama create llama3-long -f Modelfile

Use model="llama3-long" in your client calls. This avoids silent truncation on long prompts.

Mocking your LLM calls in tests

The point of a local model is a deterministic, free dev loop. Wrap the client so the base URL comes from the environment:

import os
from openai import OpenAI

def get_llm_client() -> OpenAI:
    return OpenAI(
        base_url=os.environ.get("LLM_BASE_URL", "http://localhost:11434/v1"),
        api_key=os.environ.get("LLM_API_KEY", "dev"),
    )

A pytest test that runs fully offline:

def test_chat_returns_content():
    client = get_llm_client()
    resp = client.chat.completions.create(
        model="llama3",
        messages=[{"role": "user", "content": "Return the word OK only."}],
        temperature=0,
    )
    assert "OK" in resp.choices[0].message.content

Run it with LLM_BASE_URL unset (defaults to Ollama). CI runners without GPUs can still execute this if Ollama is installed as a service.

A minimal CLI loop for manual poking:

import sys
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dev")
msgs = []
while True:
    q = input("you> ")
    if q == "/exit":
        sys.exit(0)
    msgs.append({"role": "user", "content": q})
    out = client.chat.completions.create(model="llama3", messages=msgs, stream=True)
    print("llm> ", end="")
    text = ""
    for chunk in out:
        d = chunk.choices[0].delta.content or ""
        text += d
        print(d, end="", flush=True)
    print()
    msgs.append({"role": "assistant", "content": text})

Swapping to a remote gateway

Local Llama 3 is great for iteration, but production needs redundancy and model variety. Because the client above is OpenAI-compatible, you can repoint it at a hosted gateway with zero code changes. For example, n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is degraded—set LLM_BASE_URL=https://api.n4n.ai/v1 and supply a real key, and your llama3 calls become routed, metered requests alongside Claude or Mixtral.

Keep the local Ollama setup for pre-merge tests; keep the remote for staging and prod.

Gotchas

Memory. The 8B model uses ~4.7 GB disk and ~6 GB RAM resident. Running it alongside Docker and Chrome will swap.

Cold starts. First request after pull compiles the model graph; expect a 2–5s delay. Subsequent calls are fast.

Non-determinism. Even at temperature=0, local quantization can vary outputs across hardware. Assert shape and keywords in tests, never exact strings.

API drift. Ollama’s /v1 compatibility is solid but not 100%. Function-calling schemas differ from OpenAI’s; test tool-use paths explicitly if you rely on them.

Concurrency. The default Ollama server processes one request at a time per model. Parallel tests will queue. Use OLLAMA_NUM_PARALLEL=4 env var to raise the limit if your RAM allows.

Wrapping up

You now have a reproducible way to run llama 3 locally ollama for every developer on the team. The OpenAI-compatible surface means your app code stays provider-agnostic, and your test suite can run against a real model without network egress. That’s the cheapest insurance against “works only in prod” LLM bugs.