Building reliable agentic systems means you must prompt agents self-correction retries instead of assuming a single LLM pass will succeed. Calls return malformed JSON, violate your schema, or silently misunderstand the task; the prompt and the orchestration code have to treat failure as a first-class path rather than an edge case.
Step 1: Enumerate the failure modes before writing prompts
You cannot prompt agents self-correction retries against a vague notion of “wrong.” List the concrete ways your agent can fail, and split them by retryability:
- Schema violations (missing keys, wrong types): retryable, fixable by the model.
- Tool errors (invalid arguments, downstream 4xx): retryable if the model can revise args.
- Semantic errors (plan contradicts earlier facts, no progress): retryable with feedback, but watch for loops.
- Auth/permission errors (401 from your own tool): not retryable; escalate immediately.
- Provider errors (429, 504, content filter): retryable at transport layer, not the model’s concern.
Write these into a short spec. This spec becomes the contract you enforce in both the prompt and the parser. If your agent emits a JSON action, define the required fields explicitly:
{
"action": "call_tool",
"tool": "sql_query",
"args": {"query": "SELECT 1"},
"reasoning": "string"
}Any deviation is a retryable error. Deciding retryability up front prevents you from writing a loop that hammers an unfixable condition.
Step 2: Embed correction directives in the system prompt
The model will not self-correct unless you tell it to, and tell it exactly how. A system prompt that merely describes the task yields one-shot answers. Add an explicit protocol that turns correction into a prescribed behavior:
You are a tool-calling agent. You MUST output strictly valid JSON matching the schema.
If your previous output was rejected, read the error message in the user turn and fix ONLY the broken part.
Do not apologize. Do not repeat the error explanation. Output the corrected JSON block.
You have up to 5 attempts. If you cannot satisfy the schema, output {"action":"escalate","reason":"string"}.The instruction “read the error message in the user turn” is the load-bearing line. You will inject parser errors as user messages, and the model needs permission to treat them as input rather than as a conversation violation. Also note “do not apologize”: left unchecked, models burn tokens on meta-commentary instead of the corrected payload. Keep temperature low (0–0.3) for correction turns; you want deterministic adherence to schema, not creativity.
Step 3: Force machine-readable output with response constraints
Rely on the API’s structured output features instead of nudging with “please output JSON.” With OpenAI-compatible endpoints, use response_format with a JSON schema. Below is a minimal Python snippet using the official client:
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
schema = {
"type": "object",
"properties": {
"action": {"type": "string"},
"tool": {"type": "string"},
"args": {"type": "object"},
"reasoning": {"type": "string"}
},
"required": ["action", "tool", "args", "reasoning"]
}
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"system","content":"...as above..."}],
response_format={"type":"json_schema","json_schema":{"name":"agent_cmd","schema":schema}}
)If the model lacks native schema support, fall back to {"type":"json_object"} with a strict validator. Either way, the parsing step is where you detect the need to prompt agents self-correction retries. A simple Pydantic validator makes the check explicit:
from pydantic import BaseModel, ValidationError
class AgentCmd(BaseModel):
action: str
tool: str
args: dict
reasoning: str
def validate(data: dict) -> AgentCmd:
return AgentCmd(**data) # raises ValidationError on mismatchStep 4: Implement the retry loop in your orchestrator
The prompt alone does nothing without code that catches failures and feeds them back. Write a bounded loop that separates semantic errors from transport errors:
import json
from openai import OpenAI, APIError
from pydantic import ValidationError
client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
SYSTEM = "You are a tool-calling agent... correct errors as instructed."
def run_agent(user_task: str, max_attempts: int = 5):
messages = [
{"role":"system","content":SYSTEM},
{"role":"user","content":user_task}
]
for attempt in range(max_attempts):
try:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
response_format={"type":"json_object"}
)
data = json.loads(resp.choices[0].message.content)
cmd = validate(data) # raises ValidationError on schema miss
return cmd
except (json.JSONDecodeError, ValidationError) as e:
# feed the error back as a user turn — this is the core of self-correction
messages.append({"role":"user","content":f"ERROR: {e}. Correct your output."})
except APIError as e:
# transport error: do not consume an attempt; let gateway or caller handle
raise
raise RuntimeError("agent exceeded retry budget")The messages.append with the parsed exception is the mechanical half of prompt agents self-correction retries: the model sees its mistake and gets a directive to fix it. Do not truncate the system prompt or prior context on retry; the model needs the original task to produce a coherent correction.
Step 5: Offload transport retries to an inference gateway
Semantic self-correction is your job. Transport-level resilience (429s, dead providers) should not burn your attempt budget. Point the client at an OpenAI-compatible gateway that performs automatic fallback when a provider is rate-limited or degraded. n4n.ai exposes one endpoint covering 240+ models and honors client routing directives, so a single base_url swap gives you cross-provider redundancy without extra code.
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="your-key")Your loop from Step 4 then only catches semantic/schema errors. Provider hiccups are retried upstream or routed to a healthy model. This separation keeps the agent’s self-correction focused on reasoning, not infrastructure noise. The same gateway provides per-token usage metering, so you can attribute the extra tokens burned by correction loops to a specific trace ID and tune the prompt if retries get expensive.
Step 6: Verify the loop end to end
A retry mechanism you have not tested is a liability. Verification has three layers:
Unit test the parser and feedback injection
Mock the chat completion to return malformed JSON on attempt one, valid on attempt two. Assert the loop appends the error message and returns the valid object.
def test_retry(monkeypatch):
calls = [{"content": "{bad json"}, {"content": '{"action":"escalate","reason":"x"}'}]
def fake_create(**kwargs):
return type("R", (), {"choices":[type("C", (), {"message":type("M", (), {"content": calls.pop(0)})()})()]})()
monkeypatch.setattr(client.chat.completions, "create", fake_create)
out = run_agent("do thing")
assert out.action == "escalate"Integration test with a forced fault
Run against a sandbox model and inject a schema violation via a bad tool definition. Confirm the agent escalates instead of looping forever or crashing. Track attempt_number in logs.
Production observability
Emit structured logs: attempt_number, error_type, model, latency, trace_id. Watch for agents that consistently burn 4+ attempts on the same class of error; that signals a prompt gap, not a model flaw. Gateways with per-token metering let you alert when correction tokens exceed 20% of total spend for a route.
Success means: the task completes with valid output, retries are rare (<10% of runs in healthy systems), and no unhandled APIError escapes the gateway boundary. If you prompt agents self-correction retries correctly, the agent degrades gracefully under bad input instead of crashing your pipeline—and you can prove it with the tests above.