Building an open-weight model agent gives you control over cost, data residency, and model behavior without locking into a proprietary API. This guide walks through standing up a functional agent using open-weight models such as Llama 4, Mistral, Qwen, or DeepSeek behind an OpenAI-compatible interface, so your existing SDK code works unchanged.
Step 1: Choose a model and stand up an endpoint
You have two practical paths: self-host with vLLM or TGI, or route through a gateway that already aggregates open-weight providers. Self-hosting gives maximum isolation; a gateway reduces operational burden and hides provider outages.
If you want a single OpenAI-compatible endpoint that addresses 240+ models—including the open-weight ones in this cluster—and automatically falls back when a provider is rate-limited or degraded, point your client at a service like n4n.ai. That removes the need to write your own fallback logic for every provider.
For self-hosting, launch vLLM with tool choice enabled:
vllm serve meta-llama/Llama-4-Scout --port 8000 --enable-auto-tool-choiceThen point the OpenAI client at http://localhost:8000/v1.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1", # or "https://api.n4n.ai/v1"
api_key="EMPTY_OR_REAL_KEY",
)
MODEL = "meta-llama/Llama-4-Scout" # or "mistralai/Mistral-Large", "Qwen/Qwen2.5-72B", "deepseek-ai/DeepSeek-V3", "xai/Grok-1"Picking among Llama 4, Mistral, Qwen, DeepSeek, Grok
All five publish weights and support instruction tuning. Llama 4 and Mistral-Large have the most consistent function-calling behavior in my experience. Qwen and DeepSeek handle tools well but sometimes need a stricter system prompt. Grok-1 is viable for experimentation but has a smaller ecosystem of ready-made serving templates. Choose based on your GPU budget and latency target; none of these require a proprietary account.
Step 2: Define the agent’s tools
An open-weight model agent acts by emitting structured tool calls. Define each tool as a JSON schema and map it to a Python callable.
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Return current temperature in Celsius for a city.",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City name"}
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "celsius_to_fahrenheit",
"description": "Convert a Celsius temperature to Fahrenheit.",
"parameters": {
"type": "object",
"properties": {"celsius": {"type": "number"}},
"required": ["celsius"],
},
},
},
]
def get_weather(location: str) -> str:
# stub: replace with real API call
return f"{{'temp_c': 22, 'location': '{location}'}}"
def celsius_to_fahrenheit(celsius: float) -> str:
return str(celsius * 9/5 + 32)Keep tool descriptions terse but unambiguous. Open-weight models infer intent from the schema; vague descriptions cause hallucinated arguments or skipped calls. Use concrete types and required fields wherever possible.
Step 3: Implement the agent loop
The loop sends messages, checks for tool_calls, executes them, and feeds results back. Stop when the model returns text without tools.
import json
def run_agent(user_prompt: str, max_steps: int = 5):
messages = [
{"role": "system", "content": "You are a precise agent. Use tools when needed."},
{"role": "user", "content": user_prompt},
]
for _ in range(max_steps):
resp = client.chat.completions.create(
model=MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
temperature=0,
)
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
return msg.content
for tc in msg.tool_calls:
fn = tc.function.name
args = json.loads(tc.function.arguments)
if fn == "get_weather":
result = get_weather(**args)
elif fn == "celsius_to_fahrenheit":
result = celsius_to_fahrenheit(**args)
else:
result = "error: unknown tool"
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": result,
})
return "agent did not finish"Some open-weight models emit arguments as a JSON string with escaped quotes; json.loads handles it. If you see parsing errors, enforce guided decoding in vLLM or add a small retry that asks the model to re-emit the call. Set temperature=0 for deterministic tool selection in agent loops.
Step 4: Manage context and memory
Open-weight models have finite context windows. For a long-running open-weight model agent, cap the message list and summarize old turns.
from collections import deque
class Memory:
def __init__(self, max_tokens: int = 4000):
self.max_tokens = max_tokens
self.buffer = deque()
def add(self, msg):
self.buffer.append(msg)
# crude token estimate: ~4 chars per token
while sum(len(m.get("content", "")) // 4 for m in self.buffer) > self.max_tokens:
self.buffer.popleft()
def get(self):
return list(self.buffer)Replace the inline messages list with Memory calls. For production, use the model’s actual tokenizer to count tokens. When summarizing, ask the model to compress earlier tool results into a single system note; this keeps the active context small without losing state.
Step 5: Add resilience and caching
Network failures and rate limits are normal. Wrap the model call in a retry with backoff. If you use a gateway, it may already handle provider fallback; but you should still handle 429s in your code.
import time
from openai import APIError
def chat_with_retry(**kwargs):
for attempt in range(3):
try:
return client.chat.completions.create(**kwargs)
except APIError as e:
if e.status_code == 429:
time.sleep(2 ** attempt)
else:
raise
raise RuntimeError("exhausted retries")When routing through a gateway that honors client directives, you can forward cache-control hints to the upstream provider by setting request headers. n4n.ai forwards provider cache-control hints, so you can mark static system prompts as cacheable to cut latency and cost. Do this only if your client supports extra headers.
# example with httpx, if using openai client with custom http client
# import httpx
# client = OpenAI(base_url=..., http_client=httpx.Client(headers={"x-cache-control": "ephemeral"}))Also log resp.usage on every call. Per-token metering matters when you self-host on shared GPUs or use a metered gateway; without it you cannot track agent cost per task.
Step 6: Verify the agent end to end
Write a smoke test that asserts the agent combines tool outputs correctly.
def test_agent():
out = run_agent("What is 30C in Fahrenheit and the weather in Paris?")
assert "86" in out # 30C -> 86F
assert "Paris" in out
print("PASS:", out)
if __name__ == "__main__":
test_agent()Success means the agent called both tools, received results, and produced a final natural-language answer referencing both. If the model skips a tool, tighten the system prompt or switch to a stronger open-weight model like Llama 4 or Mistral-Large.
Beyond the unit test
Run the agent against three real queries that require different tool orderings. Confirm that:
- The
tool_call_idvalues match between call and result. - The context window never overflows (watch for truncation errors).
- Retries trigger correctly when you artificially throttle the endpoint.
An open-weight model agent that passes these checks is ready for internal automation tasks. For external user-facing deployment, add input sanitization on tool arguments and a human-in-the-loop confirmation for side-effecting tools.
Operational notes
- Log every
tool_call_idand result; debugging open-weight model agent failures is mostly trace analysis. - Start with a single-loop design. A planner/executor split helps for 10+ step tasks but adds complexity.
- Track
resp.usage.total_tokensto spot runaway loops early. - If you serve multiple models, keep the tool schemas identical across them so the same agent code works when you swap
MODEL.
You now have a runnable open-weight model agent that you can point at Llama 4, Mistral, Qwen, DeepSeek, or Grok weights without rewriting your integration.