The decision between self-consistency vs self-reflection shapes how you burn tokens to make an LLM agent more reliable. Self-consistency fires many independent reasoning passes and lets the majority decide; self-reflection makes the model critique and rewrite its own work in a loop. Both improve accuracy, but they diverge hard on cost, latency, and failure modes.
What each technique actually does
Self-consistency
Self-consistency is a voting ensemble over sampled reasoning chains. You prompt the model with the same question and a chain-of-thought template, set temperature above zero, generate N completions, extract the final answer from each, and take the mode. It exploits the fact that correct reasoning paths often converge while erroneous ones scatter.
import openai
def self_consistency(question: str, n: int = 8, temperature: float = 0.7):
answers = []
for _ in range(n):
resp = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"{question}\nThink step by step."}],
temperature=temperature,
)
answers.append(parse_final_answer(resp.choices[0].message.content))
from collections import Counter
return Counter(answers).most_common(1)[0][0]The method assumes a single model call can produce varied valid reasoning. It does not require any external feedback signal, which is why it shows up in bare eval scripts.
Self-reflection
Self-reflection (e.g., Reflexion) adds a meta-step where the model evaluates its prior output and proposes revisions. Typically you run a trajectory, score it via tests, a heuristic, or the model’s own critique, then prompt the model to reflect on failures and attempt again. State persists across turns.
def self_reflect(question: str, max_iters: int = 3):
memory = []
for i in range(max_iters):
ctx = format_memory(memory, question)
draft = llm_call(ctx)
score, feedback = evaluate(draft, question)
if score >= threshold:
return draft
memory.append({
"attempt": draft,
"critique": llm_critique(draft, feedback),
})
return draft # best effortThe loop is sequential and depends on the quality of the critique prompt. It shines when the task has a verifiable signal (unit tests, SQL execution, API responses) rather than pure subjective quality.
Head-to-head dimensions
Capabilities
Self-consistency improves answers on problems with multiple valid reasoning routes—arithmetic, multi-hop QA, symbolic manipulation. It cannot recover from a model that lacks domain knowledge; if all samples are wrong, voting just picks the most confident mistake. It also does nothing for tasks where the output is a single creative artifact with no “correct” convergence.
Self-reflection handles open-ended generation where the first pass is plausibly wrong but checkable: code generation against a test suite, document drafting with style constraints, agent actions in a sandbox. It can correct factual slips when the critique has ground truth access. Unlike voting, it can incorporate new information discovered during evaluation (a stack trace, a failed HTTP call).
Cost model
Self-consistency multiplies token consumption linearly with sample count. Eight samples means roughly 8x the prompt+completion tokens of a single call, though prompt tokens are shared and can be cached if the provider honors cache-control hints. There is no extra model sophistication required, so the cost is predictable: N times a baseline.
Self-reflection adds variable overhead: each iteration costs at least one generation plus one critique call. In practice, 2–4 iterations is common, so cost is 2–5x a single shot, but only if the task fails initial validation. Successful first-try passes pay nothing extra. When metering matters, an inference gateway with per-token usage tracking (such as n4n.ai) makes the multiplier visible per route, so you can cap spend on self-consistency batches.
Latency and throughput
Self-consistency is embarrassingly parallel. If your provider allows concurrent requests, N samples finish in roughly the time of the slowest single call plus aggregation. Throughput per question drops because you consume N slots of concurrency, but wall-clock per question stays flat.
Self-reflection is serial. Total latency is sum of iteration latencies; a 3-iteration loop with 2s per call is ~6s plus critique. For interactive agents this stalls the user unless you stream intermediate states. Throughput under load is worse because each slot is occupied for the full loop duration.
Ergonomics
Self-consistency is a few lines of loop code. The hard part is answer extraction—models phrase final answers differently, so you need robust parsers or a small classifier. Once extraction works, the system is stateless and easy to test.
Self-reflection demands prompt engineering for the critique, a memory store, and termination logic. Without a clear reward signal it degrades into the model praising itself. Debugging reflection loops is painful because failures are non-deterministic across iterations and the memory grows unbounded if you don’t trim it.
Ecosystem
Both are supported in LangChain (VoteEnsemble, ReflectionAgent), LlamaIndex, and raw SDKs. Self-consistency appears in eval harnesses (e.g., lm-eval) as a standard baseline. Self-reflection is baked into agent scaffolds like AutoGPT derivatives and Microsoft’s TaskWeaver. Neither requires custom model fine-tuning; both work on any instruction-tuned checkpoint.
Limits
Self-consistency fails when reasoning variance is low (deterministic outputs) or when the model’s prior is systematically biased—all samples share the same blind spot. It also inflates cost on easy questions where one shot would suffice.
Self-reflection suffers from critique hallucinations: the model invents problems and “fixes” non-issues. It can loop indefinitely on unsolvable tasks. It requires a judge; if the judge is the same model, errors correlate and the agent becomes confidently wrong after revision.
Comparison table
| Dimension | Self-consistency | Self-reflection |
|---|---|---|
| Core mechanism | Sample N chains, majority vote | Generate, critique, revise loop |
| Best for | Math, multi-hop QA, deterministic checks | Code gen, open drafting, sandbox agents |
| Token cost | Linear multiplier (N x) | Variable (1–K x per iteration) |
| Latency | Parallelizable, N concurrent calls | Sequential, sums per iteration |
| Implementation | Simple loop + parser | Loop + memory + critique prompt |
| External signal needed | No | Yes (test, heuristic, or self-critique) |
| Failure mode | Shared blind spot across samples | Hallucinated critiques, infinite loops |
Which to choose
Batch QA or math at scale. Use self-consistency. If you have 10k questions and a tolerant SLA, fire 5–8 samples per question with high concurrency. Cache the shared prompt prefix to cut cost. Avoid reflection; the serial latency kills throughput.
Interactive coding agent. Use self-reflection with a real test runner. The agent writes code, executes, reads tracebacks, and rewrites. Self-consistency alone can’t see the stack trace. Cap iterations at three to bound cost.
Low-budget classification. Neither may be worth it. A single well-prompted call with structured output beats both unless accuracy gains are measurable. If you must, self-consistency with N=3 is cheaper to reason about than a reflection loop.
High-stakes document generation. Self-reflection with a rubric-based critique (not the model grading itself) works. Have a separate validator check citations. Self-consistency would just produce five plausible falsehoods.
Provider reliability concerns. When you scale self-consistency to hundreds of parallel samples, rate limits bite. A gateway like n4n.ai that honors client routing directives and automatically falls back when a provider degrades keeps the voting ensemble from half-completing and skewing results.
The split is clear: self-consistency vs self-reflection is a parallel-breadth vs sequential-depth trade. Pick breadth when you can verify by counting; pick depth when you can verify by checking.