Fine-tuning Qwen 3 agents for specialized workflows beats prompt engineering once you have a few hundred labeled trajectories. This tutorial builds a domain-specific agent model from Qwen3-8B-Instruct using LoRA, with reproducible data prep, training, and evaluation scripts you can run today.
Prerequisites
- A CUDA GPU with ≥24 GB VRAM (A10G, RTX 4090, or A100).
- Python 3.10+ and Git LFS installed.
- Hugging Face account with accepted Qwen3 license.
- Install dependencies:
pip install -U transformers datasets peft trl accelerate bitsandbytes
huggingface-cli login- A dataset of 200+ agent episodes in JSONL. We synthesize a small DevOps set for the demo, but real data should span failures and multi-step loops.
Dataset shape for agent trajectories
Qwen3’s chat template expects OpenAI-style tool_calls and a tools list per episode. The assistant message carries tool_calls; a subsequent tool role carries execution output.
{
"tools": [
{
"type": "function",
"function": {
"name": "kubectl",
"description": "Execute a kubectl command",
"parameters": {
"type": "object",
"properties": { "cmd": { "type": "string" } },
"required": ["cmd"]
}
}
}
],
"messages": [
{ "role": "system", "content": "You are a Kubernetes ops agent." },
{ "role": "user", "content": "Restart the api pod in namespace prod." },
{ "role": "assistant", "content": "", "tool_calls": [
{ "name": "kubectl", "arguments": { "cmd": "rollout restart deployment/api -n prod" } }
] },
{ "role": "tool", "name": "kubectl", "content": "deployment.apps/api restarted" },
{ "role": "assistant", "content": "Restarted the api deployment in prod." }
]
}Save 200+ of these to devops_agent.jsonl. Include negative examples: permission denied, wrong namespace, dry-run output. The model learns action distribution from these contrasts.
Preprocess and apply the chat template
Load with datasets and render the native template. Qwen3 tokenizes tool calls as special markers when you pass tools and messages together.
from datasets import load_dataset
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B-Instruct")
ds = load_dataset("json", data_files="devops_agent.jsonl")["train"]
def fmt(ex):
text = tok.apply_chat_template(
ex["messages"],
tools=ex["tools"],
tokenize=False,
add_generation_prompt=False,
)
return {"text": text}
ds = ds.map(fmt, remove_columns=ds.column_names)
print(ds[0]["text"][:180])Checkpoint: the printed string contains <system> and an embedded <tool_call:6124c78e> block. Sequence lengths should average <1500 tokens; if longer, trim tool outputs.
LoRA training with TRL
We use 4-bit quantization to fit 8B on 24 GB. LoRA on attention projections captures agent behavior without full fine-tune cost.
import torch
from trl import SFTConfig, SFTTrainer
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B-Instruct",
quantization_config=bnb,
device_map="auto",
)
model = prepare_model_for_kbit_training(model)
lora = LoraConfig(
r=32, lora_alpha=64, lora_dropout=0.05,
target_modules=["q_proj","k_proj","v_proj","o_proj"],
task_type="CAUSAL_LM",
)
cfg = SFTConfig(
output_dir="./qwen3-devops-agent",
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
max_seq_length=2048,
num_train_epochs=3,
learning_rate=2e-4,
logging_steps=10,
save_strategy="epoch",
)
trainer = SFTTrainer(
model=model,
args=cfg,
train_dataset=ds,
peft_config=lora,
)
trainer.train()
trainer.save_model()Expected output: training loss falls from ~1.8 to <0.5 over ~300 steps on a 200-example set. Adapter weights land in ./qwen3-devops-agent.
Run the fine-tuned agent loop
Load base + adapter and drive a minimal loop. Qwen3 emits <tool_call:6124c78e> JSON; we extract and execute against a stub.
from transformers import pipeline
import re, json
gen = pipeline(
"text-generation",
model="./qwen3-devops-agent",
tokenizer="Qwen/Qwen3-8B-Instruct",
device_map="auto",
)
tools = [{"type":"function","function":{"name":"kubectl","description":"Run kubectl","parameters":{"type":"object","properties":{"cmd":{"type":"string"}},"required":["cmd"]}}}]
def agent(user_msg):
msgs = [{"role":"system","content":"You are a Kubernetes ops agent."},
{"role":"user","content":user_msg}]
out = gen(msgs, tools=tools, max_new_tokens=256)
text = out[0]["generated_text"][-1]["content"]
m = re.search(r"<tool_call>(.*?)</tool_call>", text, re.DOTALL)
if m:
call = json.loads(m.group(1))
print("TOOL CALL:", call["name"], call["arguments"])
return "deployment.apps/api restarted"
return text
print(agent("Restart the api pod in namespace prod."))Expected output:
TOOL CALL: kubectl {'cmd': 'rollout restart deployment/api -n prod'}
deployment.apps/api restartedThe adapter steers the base model to emit your domain’s command shape instead of generic text.
Evaluate before shipping
A single happy path proves nothing. Score held-out episodes on action match and final answer.
from datasets import load_dataset
eval_ds = load_dataset("json", data_files="devops_eval.jsonl")["train"]
correct = 0
for ex in eval_ds:
pred = agent(ex["messages"][1]["content"])
if "rollout restart" in pred and "api" in pred and "prod" in pred:
correct += 1
print(f"Action accuracy: {correct/len(eval_ds):.2%}")Aim for >85% on a tight domain. If accuracy lags, inspect mispredictions—usually schema drift or missing negative samples.
Building a larger corpus synthetically
Generate variations to reach 500+ episodes without manual labeling:
import json, random
names = ["api","web","worker","cache"]
ns = ["prod","staging","dev"]
verbs = ["rollout restart deployment/{} -n {}", "scale deployment/{} -n {} --replicas=3"]
with open("devops_agent.jsonl","w") as f:
for _ in range(500):
n, s = random.choice(names), random.choice(ns)
cmd = random.choice(verbs).format(n, s)
rec = {
"tools": [{"type":"function","function":{"name":"kubectl","description":"Run kubectl","parameters":{"type":"object","properties":{"cmd":{"type":"string"}},"required":["cmd"]}}}],
"messages": [
{"role":"system","content":"You are a Kubernetes ops agent."},
{"role":"user","content": f"Fix {n} in {s}."},
{"role":"assistant","content":"","tool_calls":[{"name":"kubectl","arguments":{"cmd":cmd}}]},
{"role":"tool","name":"kubectl","content":"ok"},
{"role":"assistant","content": f"Applied {cmd}."}
]
}
f.write(json.dumps(rec)+"\n")Regenerate training data and rerun the trainer. LoRA converges faster on diverse phrasing.
Merging LoRA for single-file deploy
For production, merge adapter into base to drop Peft dependency:
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "./qwen3-devops-agent")
model = model.merge_and_unload()
model.save_pretrained("./qwen3-devops-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen3-8B-Instruct").save_pretrained("./qwen3-devops-merged")Serve the merged dir with vLLM or TGI. If you front the endpoint with an OpenAI-compatible gateway, n4n.ai can route to your Qwen 3 agent alongside 240+ models and apply per-token metering without changing client code.
Caveats
Qwen3’s tool parser is strict. Validate generated arguments against your JSON schema before execution. LoRA won’t fix fundamental reasoning gaps—curate data with negative examples (invalid commands, permission errors). Keep the base model frozen; swap adapters per tenant to isolate behavior.
Fine-tuning Qwen 3 agents is cheap enough to iterate daily. Build the data pipeline first; the training script is the easy part.