When an autonomous system goes off the rails, the root cause is rarely the model—it’s the plan. To debug AI agent planning failures, you need to treat the agent’s reasoning trace as a distributed system log, not a black box.
Step 1: Capture the full planning trace
Most agent frameworks hide the intermediate prompts behind convenience abstractions. Before you can debug AI agent planning failures, you must see every token exchanged between the planner and the executor. I’ve seen teams lose days because they only logged the final answer; the broken assumption was made in step two of a twelve-step plan and never surfaced.
Wrap your LLM client so each request and response is appended to a JSONL file. Log the system prompt, the retrieved context, the tool schemas, and the raw model output—including partial tool calls if you stream. Below is a minimal Python decorator for an OpenAI-compatible client:
import json, time, openai
log_path = "agent_trace.jsonl"
def logged_chat(client):
orig = client.chat.completions.create
def wrapper(**kwargs):
start = time.time()
resp = orig(**kwargs)
record = {
"ts": start,
"model": kwargs.get("model"),
"messages": kwargs.get("messages"),
"tools": kwargs.get("tools"),
"response": resp.model_dump(),
}
with open(log_path, "a") as f:
f.write(json.dumps(record) + "\n")
return resp
client.chat.completions.create = wrapper
return clientIf you use a framework like LangGraph or Autogen, override the model invocation layer rather than adding print statements inside nodes. The goal is a complete, ordered transcript that you can diff between runs.
Run your agent normally. Verify success by inspecting agent_trace.jsonl: you should have one line per model call, including the system prompt that instructs the planner, the user goal, and the raw tool-call arguments. If a line is missing the tools field but the agent clearly used a tool, your instrumentation is incomplete.
Step 2: Reproduce the failure in isolation
A planning failure that only shows up after 12 steps of interaction is impossible to iterate on. Extract the exact message list that produced the bad plan and replay it with temperature 0. Crucially, mock any external tool side effects so you are not sending emails or deleting rows while debugging.
import json, openai
records = [json.loads(l) for l in open("agent_trace.jsonl")]
bad = records[3] # example index of first suspect plan
client = openai.OpenAI()
resp = client.chat.completions.create(
model=bad["model"],
messages=bad["messages"],
tools=bad["tools"],
temperature=0,
seed=42,
)
print(resp.choices[0].message)If the same broken plan appears, you have a deterministic repro. If it diverges, lower temperature further or check for non-deterministic tool outputs feeding back into the context. For agents that interleave planning with execution, snapshot the state right before the faulty plan step and serialize it. Then write a tiny script that loads that state and calls the planner.
Verify success when the offending plan is reproducible across three consecutive runs and you can trigger it without executing any downstream tools.
Step 3: Validate task decomposition boundaries
When you debug AI agent planning failures, decomposition boundaries are the first thing to audit. Planning agents fail when a sub-task is vague or unbounded. “Analyze the dataset” is not a step; it’s a project. Load the planner’s output and assert each step has a goal and an explicit completion criterion. Forcing the model to emit done_when strings reduces goal drift by making the executor’s return signal unambiguous.
def check_plan(plan):
assert isinstance(plan, list), "plan must be a list of steps"
for i, step in enumerate(plan):
assert "goal" in step, f"step {i} missing goal"
assert "done_when" in step, f"step {i} missing done_when"
assert len(step["goal"].split()) < 30, f"step {i} goal too broad"
return True
plan = [
{"goal": "fetch user data", "done_when": "have JSON with id"},
{"goal": "summarize", "done_when": "100-word summary in state"},
]
check_plan(plan)Run this against the captured planner response. Don’t stop at structural checks. Manually read the done_when clauses: if they reference state that the executor cannot observe, the plan is still broken. For hierarchical agents, verify that sub-plans spawned by a sub-agent also satisfy the same schema.
Verify success by confirming the checker raises on the actual failing plan, then passes after you manually edit the plan to include done_when fields. Then feed the edited plan to your executor and confirm it actually terminates.
Step 4: Check tool and function schema mismatches
A planner will happily emit {"user_id": "alice"} when the tool expects {"uid": 123}. These mismatches surface later as execution errors but originate in planning. Use JSON Schema to validate every tool call the planner produces. Keep your schemas in versioned files; schema drift between the agent prompt and the actual API is a top cause of silent planning rot.
{
"name": "get_user",
"parameters": {
"type": "object",
"properties": {
"uid": {"type": "integer", "minimum": 1}
},
"required": ["uid"]
}
}import jsonschema
def validate_call(call, schema):
jsonschema.validate(instance=call["arguments"], schema=schema["parameters"])
bad_call = {"name": "get_user", "arguments": {"user_id": "alice"}}
try:
validate_call(bad_call, schema)
except jsonschema.ValidationError as e:
print("schema violation:", e.message)Beyond type checks, validate semantic constraints: if uid must be positive, say so in the schema. The model does not know your database internals.
Verify success by running the validator over all tool calls in your trace; you should see at least one validation error that correlates with the agent’s later confusion. If the call passes validation but still fails at runtime, the schema is lying about the contract.
Step 5: Swap models to isolate planner competence
Sometimes the plan is structurally fine but the chosen model lacks the reasoning budget. Route the same prompt through a stronger or weaker model to see if the failure is model-specific. If you route through n4n.ai, you can flip the planner model by changing one header and get automatic fallback when a provider is degraded, without rewriting agent code. This is also useful when the primary provider is rate-limiting your debug runs.
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "x-model: anthropic/claude-3.5-sonnet" \
-d '{
"model": "openai/gpt-4o",
"messages": [{"role":"system","content":"You are a planner."},
{"role":"user","content":"Migrate DB schema"}]
}'The gateway honors the client routing directive and forwards the request to the specified model. It also forwards provider cache-control hints, so repeating the same planner prompt across experiments hits cache instead of burning tokens.
Verify success by comparing the plan from the alternate model against the original; if the new plan is correct, your debug AI agent planning failures effort has pinpointed model selection as the culprit. If both models fail identically, the bug is in your prompt or schema, not the weights.
Step 6: Encode the fix as a regression test
Debugging is not done until the failure cannot silently return. Write a pytest that loads the captured failing messages, runs the planner, and asserts the plan passes check_plan and all tool calls validate. Store the trace as a fixture so CI runs the exact repro.
def test_planner_no_bad_calls():
records = load_trace("agent_trace.jsonl")
bad = records[3]
plan = extract_plan(bad)
check_plan(plan)
for call in extract_calls(bad):
validate_call(call, TOOL_SCHEMAS[call["name"]])
def test_validator_fires():
import pytest
with pytest.raises(jsonschema.ValidationError):
validate_call({"name":"get_user","arguments":{"user_id":"x"}}, schema)Add the second test to ensure your validator actually fires. I’ve been burned by assertions that passed because the validation function swallowed exceptions.
Run pytest test_planner.py. Verify success when the test fails on the original trace but passes after you apply the prompt or schema fix that emerged from steps 3–5. At that point, you have converted a mysterious planning outage into a guarded invariant.
Treat planning as a testable component. The next time you debug AI agent planning failures, you will already have the harness wired in.