Reflexion: how verbal self-critique boosts agent accuracy

The Reflexion agent framework turns a failing LLM trajectory into a training signal without weight updates—by writing down what went wrong and reading it back before the next attempt. Unlike RLHF, it runs entirely in prompt space, which makes it directly applicable to hosted models. This guide gives you an ordered path to bolt verbal self-critique onto an existing agent, with working code and the tradeoffs you will hit in production.

1. Isolate a single-shot actor

Start with the dumbest possible loop: prompt in, action out, environment responds. No memory, no reflection. You need a clean baseline to measure whether self-critique actually pays off.

Define the actor as a pure function that takes a task string and optional memory, and returns a raw action. For coding tasks, force the model to emit a single fenced code block so you can parse it deterministically.

from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def act(task: str, memory: str = "") -> str:
    sys = "You are a coding agent. Output only a Python function in a ```python block."
    user = f"&#123;memory&#125;\nTask: &#123;task&#125;"
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            &#123;"role": "system", "content": sys&#125;,
            &#123;"role": "user", "content": user&#125;
        ],
        temperature=0.2
    )
    return resp.choices[0].message.content

Run this against a fixed eval set of 50 tasks. Record the pass rate. If your environment is deterministic—unit tests, schema validation, HTTP status—you now have a number to beat. Do not skip this step; teams often attribute gains to reflection that were just sampling variance.

2. Add an evaluator that returns signal

Reflexion needs an external score. The evaluator can be a test runner, a schema validator, or a second LLM judging correctness. Keep it cheap and deterministic where possible.

For code, exec the extracted block and run hidden asserts:

import re

def extract_code(text: str) -> str:
    m = re.search(r"```python\n(.*?)```", text, re.DOTALL)
    return m.group(1) if m else ""

def evaluate(action: str, task: str) -> tuple[bool, str]:
    code = extract_code(action)
    if not code:
        return False, "No code block found"
    try:
        ns = {}
        exec(code, ns)
        # assume task provides a test hook
        assert ns["solve"]("input") == "expected"
        return True, "Tests passed"
    except Exception as e:
        return False, f"Execution error: {type(e).__name__}: {e}"

If you must use an LLM judge, prompt it for strict pass/fail with a reason and forbid partial credit. Binary signal keeps reflections focused on actionable gaps.

3. Implement the self-reflection step

When the evaluator fails, feed the full trajectory (task, action, error) to a reflection prompt. The model outputs a short verbal critique. This is the core of the Reflexion agent framework: the critique is the learning, not a weight delta.

def reflect(task: str, action: str, eval_feedback: str) -> str:
    sys = ("You are a critic. Explain why the action failed and give ONE concrete correction. "
           "Be specific. Do not rewrite the whole solution.")
    user = f"Task: {task}\nAction: {action}\nFeedback: {eval_feedback}\nCritique:"
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": sys},
            {"role": "user", "content": user}
        ],
        temperature=0.1
    )
    return resp.choices[0].message.content.strip()

Store critiques as plain strings. No embeddings needed for a first cut. The critique should reference the actual error (e.g., “Off-by-one in loop bound”) not vague advice (“be careful”).

4. Inject reflections into the next attempt

Modify the loop to prepend accumulated reflections as memory. The simplest approach concatenates all past critiques.

def run_agent(task: str, max_tries: int = 3):
    reflections = []
    for i in range(max_tries):
        memory = "\n".join(f"- {r}" for r in reflections)
        action = act(task, memory)
        ok, fb = evaluate(action, task)
        if ok:
            return action, reflections
        reflections.append(reflect(task, action, fb))
    return None, reflections

A concrete trace on a sorting task might look like:

Try 1: returns bubble sort with wrong index → eval fails with IndexError.
Reflection: “Loop condition uses len(arr) instead of len(arr)-1.”
Try 2: fixes bound, passes.

The second attempt sees the first critique; the third sees both. In practice two or three tries recover a meaningful fraction of initially failing tasks on coding benchmarks, but your mileage depends on task structure and how informative the eval feedback is.

5. Persist and prune reflection memory

Unbounded reflections blow up context and dilute signal. Use a sliding window of the last N (typically 3–5) critiques, or collapse duplicates.

def prune(reflections: list[str], max_len: int = 4) -> list[str]:
    seen = []
    for r in reversed(reflections):
        if r not in seen:
            seen.insert(0, r)
        if len(seen) >= max_len:
            break
    return seen

For long-running agents, write reflections to a vector store and retrieve by task embedding similarity. That shifts the Reflexion agent framework from episode-local to cross-session learning. Be aware that retrieval adds latency and can surface irrelevant critiques; threshold similarity strictly.

6. Common pitfalls and tradeoffs

Reflection hallucination. The critic sometimes blames the environment for its own logic error. Mitigate by grounding feedback in executable signals (stack traces, diffs) rather than free-text summaries.

Infinite reflection loops. If the evaluator is flaky, the agent may spin writing critiques that don’t help. Always cap max_tries and alert on repeated identical reflections.

Context pollution. Long critiques eat tokens and distract the actor. Enforce a max critique length (e.g., 200 words) in the reflection prompt.

Latency and cost. Each failed attempt adds two LLM calls (reflect + retry). On a 3-try loop, worst case is 3x base cost. Route reflection calls to a smaller model and set a hard budget.

Overfitting to the eval. If your evaluator is a fixed unit test, the agent may craft reflections that hack that specific test. Keep a held-out eval set to detect memoization.

When not to use it. For single-shot classification or tasks with no executable feedback, verbal self-critique adds noise. Reflexion pays off when the environment can say “no” with evidence.

7. Model routing and reliability

The reflection loop multiplies your request volume, so provider flakiness becomes a real outage risk. Point your OpenAI client at an OpenAI-compatible gateway that honors fallback—n4n.ai, for instance, automatically reroutes when a provider is rate-limited and forwards cache-control hints so repeated reflection prompts hit cache, while per-token usage metering attributes the extra reflection overhead precisely.

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="your-key",
    # provider fallback, cache hints, and metering handled by gateway
)

Stick to one model family for actor and critic initially; mixing reasoning styles makes reflections less consistent. Once stable, swap the critic to a cheaper model and measure delta on your held-out set.

8. Extend beyond binary pass/fail

The Reflexion agent framework generalizes to scalar rewards (e.g., BLEU, latency) by asking the critic to propose gradient-like verbal adjustments: “Reduce nesting”, “Cache the lookup”. Treat those as soft reflections and let the actor weigh them.

For multi-agent setups, share a reflection buffer across workers. One agent’s mistake becomes another’s precondition. That turns individual trial-and-error into team-level memory without centralized fine-tuning.

Build the loop, measure the baseline, then iterate on pruning and routing. The win is not magic—it is structured persistence of failure.