Most agent frameworks ship with a default reflex: wrap every model call in a retry loop and append a “verify your answer” step. This pattern treats agent retries self-correction latency as an acceptable cost of reliability, but the tax is heavier than teams realize—redundant generation and verification passes routinely double end-to-end time without addressing the actual failure modes. The seconds disappear into duplicate inference, not network jitter.
The anatomy of a naive retry
The typical implementation is a decorator that catches any exception and tries again. It is simple to write and seductive to keep:
from tenacity import retry, stop_after_attempt, wait_fixed
@retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
def call_llm(prompt):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
return resp.choices[0].message.contentThis code does not distinguish a transient 429 from a permanent schema violation. If the prompt reliably produces a string when you expected JSON, you pay for three full round trips and three generations. The wait adds dead time on top.
Worse, the retry wraps the entire agent step, so any downstream validation failure triggers a rebuild of the same context. In a multi-tool agent, that context can be several thousand tokens. Re-sending it multiplies both latency and cost.
Where the seconds actually go
Break a single LLM call into two phases: time to first token (TTFT) and generation time. For a mid-size model on a crowded endpoint, TTFT of 500 ms–2 s is normal; generation of a 300-token response might take 2–5 s. A retry on a permanent error adds another full TTFT + generation. Self-correction adds a second call where the input includes the prior output, so the input is larger and the generation is often longer because the critique instructions bloat the prompt.
Consider an agent that writes a SQL query, then asks the model to review it:
{
"step": "generate_sql",
"output": "SELECT * FROM users WHERE id = '1'",
"self_correction_prompt": "Review the SQL for syntax errors and security issues, then output corrected SQL."
}The second call is a complete forward pass. There is no free lunch—the GPU computes new tokens from scratch. If the first call took 3 s, the correction step typically adds 3–6 s because the context grew. Agent retries self-correction latency is therefore dominated by redundant compute, not by the occasional network timeout.
Self-correction is a second inference, not a debugger
A model critiquing its own output has no special access to ground truth. Unless you feed it execution results—a unit test, a database error, a type checker—it is guessing based on patterns in the training data. The loop below is common and expensive:
def agent_step(obs):
plan = llm(obs)
critique = llm(f"Critique this plan: {plan}")
if "error" in critique:
plan = llm(f"Fix the plan given critique: {critique}")
return planThree generations per step. In a ten-step agent, that is up to thirty model calls where ten might have sufficed. The latency compounds linearly with step count, and the marginal reliability gain is unverified.
When retries earn their keep
Retries are not evil; they are misapplied. Transient provider errors—HTTP 429, 503, occasional gateway timeouts—are legitimate retry targets. The problem is client-side retry storms that hammer a degraded provider and worsen the outage. An OpenAI-compatible gateway such as n4n.ai handles automatic fallback when a provider is rate-limited or degraded, which lets you drop client retry logic for transient provider errors and focus retries only on application-level validation.
Application-level retries make sense when the failure is correctable with feedback. A JSON parse error is a good example: the first attempt produced a string, you parse, it fails, you retry with the error message appended. That is a single targeted retry, not a blind loop.
def extract_json(text):
for attempt in range(2):
try:
return json.loads(llm(text))
except json.JSONDecodeError as e:
text = f"{text}\nPrevious output was invalid: {e}. Return only JSON."
raise ValueError("could not parse")Here the retry carries specific signal. That is defensible.
Measuring the latency tax
You cannot optimize what you do not measure. Wrap your model calls with a lightweight probe and record percentiles:
import time, functools, statistics
def latency_probe(fn):
@functools.wraps(fn)
def wrapper(*a, **k):
t0 = time.perf_counter()
res = fn(*a, **k)
dt = time.perf_counter() - t0
wrapper.samples.append(dt)
return res
wrapper.samples = []
return wrapper
# after workload
p95 = statistics.quantiles(latency_probe.samples, n=20)[18]Run a representative task with and without the self-correction step. Engineers often underestimate agent retries self-correction latency because they profile only the happy path. The p95 with correction enabled frequently doubles, and the tail stretches because retries stack during provider slowdowns.
Token math: the hidden multiplier
Every retry and correction pass re-sends the input context and generates new output tokens. If your agent step consumes 2k input tokens and emits 500 output tokens, a single retry doubles that to 4k in / 1k out. Self-correction typically feeds the prior output back, so the second call might be 2.5k in / 400 out. The cumulative token volume scales with the number of passes, and most inference billing is per token. Even without quoting prices, the slope is clear: more passes, more spend, more latency.
A concrete workflow: document extraction
Take an invoice extraction agent. The naive version retries the whole call on any missing field and then runs a “self-check” prompt:
def naive_extract(pdf_text):
for _ in range(3):
raw = llm(f"Extract invoice fields: {pdf_text}")
if "invoice_id" in raw and "total" in raw:
return raw
return llm("Fix your previous extraction") # open-ended correctionThe better version validates with a schema and retries once with the exact error:
from pydantic import BaseModel, ValidationError
class Invoice(BaseModel):
invoice_id: str
total: float
def tuned_extract(pdf_text):
ctx = pdf_text
for _ in range(2):
try:
return Invoice.model_validate_json(llm(ctx))
except ValidationError as e:
ctx = f"{pdf_text}\nValidation failed: {e}. Return valid JSON."
raise RuntimeError("extraction failed")The tuned version caps generations at two, sends precise feedback, and skips the vague self-correction call. In practice this cuts agent retries self-correction latency by half while improving parse success because the model gets actionable signal.
Strategies to cut the latency tax
Separate transient from permanent
Catch RateLimitError and APIError differently from ValueError. Only the former should trigger a blind retry, and preferably via gateway fallback.
Use constrained decoding
If your endpoint supports JSON mode or grammar constraints, request them. This removes an entire class of parse retries.
Limit self-correction to verifiable steps
Only invoke a critique pass when you can run the output—execute the code, call the tool, check the type. Otherwise skip it.
Parallelize independent calls
If the agent has two independent sub-tasks, fan them out concurrently instead of serial retry-with-correction.
Honor cache hints
When a gateway forwards provider cache-control hints, structure prompts so stable prefixes are cached. Retries then hit warm caches and reduce TTFT.
Decisive takeaway
Treat agent retries self-correction latency as a designed parameter, not a side effect. Default to zero retries for logic and schema errors; use a single targeted retry with explicit validation feedback when needed. Offload transient provider failures to an inference gateway with automatic fallback so your client code stays thin. Reserve open-ended self-correction for steps where you can execute the result and confirm the fix. Agents built this way respond in seconds, not tens of seconds, and waste fewer tokens on guesses.