Tree of Thoughts vs Reflexion: which self-correction wins

Tree of thoughts vs reflexion is the comparison every engineer building self-correcting agents eventually hits. Both promise better answers than a single greedy decode, but they spend compute in opposite ways: one searches a tree, the other loops with memory.

What each approach actually does

Tree of Thoughts

Tree of Thoughts (ToT) frames reasoning as a search over a directed graph of intermediate “thoughts.” You decompose a problem into steps, generate multiple continuations at each step, and use the model itself (or a heuristic) to score partial solutions. A search algorithm—BFS, DFS, or beam—keeps the most promising branches.

The win is explicit exploration. For tasks with a verifiable structure (game states, math constraints, puzzle moves), ToT finds paths a left-to-right decoder would prune too early.

Reflexion

Reflexion skips the tree. An agent acts in an environment, gets a scalar or textual reward, and on failure asks the model to write a natural-language reflection about what went wrong. That reflection is stored in episodic memory and injected into the next attempt’s prompt. No branching search; just retry with better context.

The win is cheap introspection. The model learns from its own mistakes within a single session without weight updates.

Head-to-head dimensions

Capabilities

ToT excels when the space of solutions is combinatorial and intermediate states can be evaluated cheaply. It will systematically cover angles Reflexion might never stumble into.

Reflexion excels when the task gives a clear signal (unit tests, SQL errors, human thumbs-down) and the fix is usually a local correction. It naturally handles long-horizon agent loops where you cannot define a thought grammar upfront.

Cost model

ToT multiplies token spend by branch_factor ^ depth if you naively expand, though beam search caps it. Scoring nodes also costs tokens. A 3-branch, 4-depth beam with evaluator calls can be 20–50x a single completion.

Reflexion adds one reflection completion per failed trial plus the retried trajectory. If your task succeeds in 1–2 tries, it is far cheaper. If it loops 10 times, cost converges with a small ToT.

When running either pattern at scale, an OpenAI-compatible gateway such as n4n.ai that provides per-token metering and automatic fallback across 240+ models lets you route expansion or reflection calls to cheaper checkpoints without rewriting your loop.

Latency and throughput

ToT can parallelize branch evaluation across async requests, but worst-case wall-clock is the deepest path times node latency. Reflexion is strictly sequential: each failure adds a full round-trip. For interactive UX, ToT with a tight beam feels more predictable; Reflexion feels like a thinking user—pauses, then answers.

Ergonomics

ToT forces you to answer hard design questions: what is a thought, how do you score it, what search budget is allowed. Get those wrong and you burn money for no accuracy gain.

Reflexion needs an environment that returns feedback and a memory store. The reflection prompt is the only novel surface; the rest is a normal agent loop.

Ecosystem and tooling

LangChain ships TreeOfThoughts and several paper repos exist. Reflexion has a reference implementation and is baked into agent scaffolds like Autogen and some LlamaIndex patterns. Neither is turnkey in production; both assume you own the orchestration.

Limits

ToT inherits LLM self-evaluation bias: the scorer is the same model that wrote the branch, so confident wrong answers survive. Reflexion can get stuck in a reflection loop, re-deriving the same mistake with fancier words. Both degrade when the base model is weak—search or reflection cannot fix a model that can’t do the task.

Comparison table

Implementation sketches

A minimal ToT expansion in Python:

def expand(state: str, k: int = 3) -> list[str]:
    # call LLM to produce k continuations
    return [llm_completion(state) for _ in range(k)]

def score(state: str) -> float:
    # model-as-judge or heuristic
    return float(llm_completion(f"score 0-1: {state}") )

def tree_of_thoughts(root: str, depth: int = 3, beam: int = 3):
    frontier = [root]
    for _ in range(depth):
        candidates = []
        for s in frontier:
            for child in expand(s, beam):
                candidates.append((child, score(child)))
        frontier = [s for s, _ in sorted(candidates, key=lambda x: -x[1])[:beam]]
    return frontier[0]

A minimal Reflexion loop:

memory = []
for trial in range(5):
    traj = agent_run(task, memory)
    reward, feedback = env.evaluate(traj)
    if reward >= 1.0:
        break
    reflection = llm_completion(
        f"Task failed: {feedback}\nTrajectory: {traj}\nWhat went wrong?"
    )
    memory.append(reflection)

Both snippets omit retries, parsing, and provider config, but show the shape difference.

Which to choose

Use Tree of Thoughts when

• The problem has discrete steps and a cheap verifier (compiler, math check, rule engine).
• You can define a thought unit smaller than a full answer.
• Accuracy matters more than per-request cost, and you can cap breadth.

Example: generating a multi-step SQL query where each clause can be validated against a schema.

Use Reflexion when

• The environment already returns pass/fail or error text (code exec, API calls, chat rating).
• Tasks are open-ended and you cannot pre-specify a search grammar.
• You want minimal new infrastructure beyond a prompt and a list.

Example: an agent that writes a script, runs it, reads the traceback, and fixes it.

Hybrid notes

In practice, many production systems blend them: a coarse ToT to pick a strategy, then Reflexion inside the chosen branch. Keep the tree shallow (depth 2, beam 2) and let reflection handle local fixes. That bounds cost while retaining recovery from blind spots.

If you only ship one, ship Reflexion first—it is easier to instrument and the failure signal is usually already in your logs.