Run Llama 4 Maverick locally with Ollama and LangChain

If you want to keep inference data on your own metal and avoid per-token API pricing, local models are the obvious move. To run Llama 4 Maverick with Ollama LangChain, you only need a working Ollama daemon and the lightweight langchain-ollama package. This guide walks through each step with copy-pasteable commands and a verification check at the end.

Step 1: Install Ollama

Ollama ships a single static binary for Linux and a packaged app for macOS. On Linux, the official install script drops the binary and enables a systemd user service:

curl -fsSL https://ollama.com/install.sh | sh

On macOS, grab the .dmg from the Ollama site or use Homebrew:

brew install ollama

Windows users should install WSL2 Ubuntu and run the Linux script there—native Windows builds exist, but WSL2 gives you the mature CUDA passthrough path. After install, confirm the daemon responds:

ollama --version

Expect a version string like ollama version 0.5.x. If you see command not found, your PATH missed the install location; source your shell rc or reopen the terminal.

Step 2: Pull the Llama 4 Maverick model

Ollama distributes models as tagged images. The Maverick variant of Llama 4 is published under the llama4-maverick namespace (run ollama search llama4 if the tag changed). Pull it explicitly:

ollama pull llama4-maverick

The download size depends on the quantization Ollama selects. For a 32B-class model, the default tag usually fetches a q4_K_M or similar GGUF that fits in roughly 20–24 GB of VRAM or system RAM. If you’re on a smaller machine, append a specific quant tag, e.g. llama4-maverick:q3_K_L, if available.

While pulling, Ollama prints layer hashes and a progress bar. When it returns to the prompt, the model is stored in ~/.ollama/models. If you later need a larger context window, write a Modelfile:

FROM llama4-maverick
PARAMETER num_ctx 8192

Then build a custom tag: ollama create my-maverick -f Modelfile. For a first run, the default config is fine.

Step 3: Verify the model serves locally

Before touching Python, run a raw inference call through the Ollama CLI to confirm the weights load and the backend works:

ollama run llama4-maverick "What is the capital of Iceland?"

You should see a coherent answer (Reykjavík) within a few seconds on a GPU, or slower on CPU-only. If you get Error: model not found, the pull didn’t finish. If you get a CUDA out-of-memory error, drop to a lower quant or free GPU memory.

This step isolates model issues from LangChain issues later. You can also hit the REST API directly:

curl http://localhost:11434/api/generate -d '{"model":"llama4-maverick","prompt":"hi"}'

A JSON object with a response field confirms the daemon is listening on 11434.

Step 4: Set up a Python environment

Keep local LLM experiments in an isolated venv. Python 3.10+ is safe for the current LangChain stack.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

Avoid installing LangChain into your system Python; dependency conflicts with pydantic and httpx are common and annoying. Check your interpreter with which python before proceeding.

Step 5: Install LangChain Ollama integration

The monolithic langchain package no longer bundles model connectors. Install the Ollama-specific extra:

pip install langchain-ollama

This pulls langchain-core and a thin ChatOllama client that speaks Ollama’s native REST API. If you also want prompt templates and memory helpers, add langchain itself:

pip install langchain

Pin versions in a requirements file for reproducibility, e.g. langchain-ollama==0.2.*. The API surface we use below is stable across recent 0.2 releases.

Step 6: Write a minimal chat script

The simplest way to run Llama 4 Maverick with Ollama LangChain is a synchronous invoke. Create basic_chat.py:

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="llama4-maverick",
    temperature=0.7,
    # Ollama runs locally; no API key needed
)

response = llm.invoke("Explain the difference between a mutex and a semaphore.")
print(response.content)

Run it:

python basic_chat.py

The script blocks until the model finishes generating. response is an AIMessage; .content holds the string. If you get ConnectionRefusedError, Ollama isn’t running—start it with ollama serve in another terminal.

For async callers, the same class exposes ainvoke:

import asyncio
from langchain_ollama import ChatOllama

async def main():
    llm = ChatOllama(model="llama4-maverick")
    resp = await llm.ainvoke("Ping")
    print(resp.content)

asyncio.run(main())

Step 7: Stream tokens and manage context

For interactive apps, streaming is non-negotiable. ChatOllama supports the standard LangChain stream interface:

from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama4-maverick", temperature=0.5)

for chunk in llm.stream("Write a 3-line bash script that backs up /etc."):
    print(chunk.content, end="", flush=True)
print()

The async variant astream works identically inside a coroutine. To add a system prompt and multi-turn context, use ChatPromptTemplate with a simple in-memory list:

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, AIMessage

llm = ChatOllama(model="llama4-maverick")
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a terse senior SRE. Answer in bullet points."),
    ("placeholder", "{history}"),
    ("human", "{input}"),
])

history = []
while True:
    user_input = input("> ")
    if user_input.lower() in {"exit", "quit"}:
        break
    chain = prompt | llm
    resp = chain.invoke({"history": history, "input": user_input})
    print(resp.content)
    history.append(HumanMessage(user_input))
    history.append(AIMessage(resp.content))

This loop keeps the last N messages in process memory. For production, swap the list for a persistent store like Redis with langchain_community retrievers. Recent Ollama builds also return usage_metadata on the response; inspect resp.usage_metadata to see prompt and completion token counts.

Step 8: Verify success and troubleshoot

Success criteria: the script prints coherent, task-appropriate text, and streaming outputs tokens incrementally without errors. Run the streaming example and watch for:

• No import errors – confirms langchain-ollama is installed correctly.
• No connection errors – confirms Ollama daemon is up on 11434.
• Sensible output – confirms the model weights are intact and the quant isn’t degenerate.

Common failure modes:

1. model not found from LangChain – Ollama tag mismatch. Run ollama list and copy the exact name into ChatOllama(model=...).
2. Garbled output – you pulled a too-aggressive quant (e.g., q2). Re-pull a higher-quality tag.
3. Sluggish responses – CPU fallback. Run nvidia-smi (or rocm-smi) to confirm Ollama is using the GPU. Set CUDA_VISIBLE_DEVICES if you have multiple GPUs.
4. Hang on first call – Ollama is lazy-loading weights; wait or prewarm with ollama run once.

If you later need to call the same code against a remote gateway, swap ChatOllama for ChatOpenAI with a compatible base_url—the LangChain abstraction makes that a one-line change.

Step 9: Hardware and quantization notes

Llama 4 Maverick is not a toy model. Even quantized, it demands real memory bandwidth. On a 16 GB MacBook, expect single-digit tokens/sec with the q4 variant; on a 24 GB RTX 4090, expect multiples of that. These are order-of-magnitude observations from similar-class open weights, not measured benchmarks—your mileage varies with context size and batch settings.

Ollama exposes advanced knobs via Modelfile if you need to bump num_ctx or set num_gpu. For most local dev, the defaults are fine. Keep the Ollama port bound to localhost; if you must expose it, put it behind an authenticated reverse proxy.

Beyond the basics

You now have a reproducible path to run Llama 4 Maverick with Ollama LangChain, from bare metal to a streaming Python REPL. The same pattern extends to other open weights—swap the model tag and you’re talking to Mistral, DeepSeek, or Qwen with zero code changes beyond the constructor argument. Build a small CLI or FastAPI wrapper around the chain object, add a retriever for RAG, and you’ve got a fully local assistant that respects your data boundaries.