Chain-of-thought prompting: a step-by-step guide

Chain-of-thought prompting forces a model to show its work before producing an answer. This chain of thought prompting guide walks through the mechanics, when to use it, and how to implement it reliably in production systems. You’ll see concrete patterns for zero-shot, few-shot, and structured reasoning — plus the failure modes that bite teams at scale.

What chain-of-thought actually does

Standard prompting asks for an answer. Chain-of-thought prompting asks for the reasoning trace that leads to the answer. The model emits intermediate tokens that represent steps — arithmetic, logical deduction, code planning, constraint checking — before the final token.

The mechanism is simple: the autoregressive nature of transformers means each token conditions on all previous tokens. By forcing the model to generate reasoning tokens first, you change the conditional distribution for the answer token. The model “commits” to a reasoning path before concluding.

This works because next-token prediction on reasoning traces correlates with correct final answers in the training data. The model has seen countless examples of “step 1, step 2, therefore answer” in code, math, and explanatory text. Prompting for the trace activates that pattern.

Zero-shot chain-of-thought

The simplest form: append “Let’s think step by step” or a more specific instruction to your prompt. No examples required.

ZERO_SHOT_COT = """{problem}

Let's think step by step."""

This works surprisingly well for arithmetic, multi-hop reasoning, and code generation. The model generates a plausible reasoning trace, then conditions the answer on it.

Tradeoff: you get no control over the reasoning structure. The model may skip steps, hallucinate intermediate values, or produce verbose but irrelevant text. For production, you’ll want more constraint.

Few-shot chain-of-thought

Provide 3–8 examples of (problem, reasoning, answer) triples. This teaches the model your preferred reasoning style, notation, and stopping condition.

FEW_SHOT_COT = """Problem: Roger has 5 tennis balls. He buys 2 more cans of 3 balls each. How many balls does he have now?
Reasoning: Roger starts with 5 balls. Each can has 3 balls. 2 cans × 3 balls = 6 new balls. 5 + 6 = 11 balls.
Answer: 11

Problem: A train leaves Chicago at 60 mph. Another leaves Detroit at 80 mph. They're 280 miles apart. When do they meet?
Reasoning: Combined speed = 60 + 80 = 140 mph. Time = distance / speed = 280 / 140 = 2 hours.
Answer: 2 hours

Problem: {problem}
Reasoning:"""

Key design decisions:

• Keep examples diverse but consistent in reasoning depth
• Match the domain of your production queries
• Use a clear delimiter (“Reasoning:”, “Answer:”) so you can parse the output programmatically
• Limit total context — few-shot eats tokens fast

Pitfall: examples that are too similar cause the model to pattern-match superficially. Examples that are too diverse confuse the reasoning style. Curate deliberately.

Structured chain-of-thought

For production systems, unstructured reasoning text is hard to parse, validate, or debug. Structure the trace as JSON, XML, or a custom schema.

STRUCTURED_COT = """{problem}

Think through this step by step. Output your reasoning as JSON with this schema:
{
  "steps": [
    {"step": 1, "description": "...", "calculation": "...", "result": "..."}
  ],
  "final_answer": "..."
}

Only output valid JSON. No extra text."""

Example output:

{
  "steps": [
    {"step": 1, "description": "Calculate total new balls", "calculation": "2 * 3", "result": "6"},
    {"step": 2, "description": "Add to initial count", "calculation": "5 + 6", "result": "11"}
  ],
  "final_answer": "11"
}

Benefits:

• Programmatic validation of each step
• Easy to strip reasoning for the final user-facing answer
• Enables step-level logging, evaluation, and intervention
• Forces the model into a predictable format

Cost: more tokens, stricter formatting requirements, occasional JSON parsing failures. Use a repair loop:

def parse_structured_cot(response: str, max_retries: int = 2) -> dict:
    for attempt in range(max_retries + 1):
        try:
            return json.loads(response)
        except json.JSONDecodeError:
            if attempt == max_retries:
                raise
            # Ask model to fix formatting
            response = call_model(f"""The previous response was invalid JSON. Fix it:
{response}

Output only valid JSON matching the schema.""")

Self-consistency: majority vote over multiple traces

Single traces can be unlucky. Self-consistency runs the same prompt N times (temperature > 0), extracts answers, and takes the majority vote.

async def self_consistency(prompt: str, n: int = 5, temperature: float = 0.7) -> str:
    tasks = [call_model(prompt, temperature=temperature) for _ in range(n)]
    responses = await asyncio.gather(*tasks)
    
    answers = [extract_answer(r) for r in responses]
    # Majority vote
    return max(set(answers), key=answers.count)

This improves accuracy on math and reasoning benchmarks significantly. Tradeoff: N× latency and cost. In production, use it selectively — high-stakes queries, ambiguous problems, or as a background verification pass.

Practical tip: cache the prompt hash. If you’ve already run self-consistency for an identical prompt, reuse the result.

Tree-of-thought and graph-of-thought: when linear isn’t enough

Chain-of-thought is linear. Some problems need branching, backtracking, or parallel exploration. Tree-of-thought (ToT) maintains multiple reasoning paths, evaluates them, and prunes.

Simplified ToT loop:

class TreeOfThought:
    def __init__(self, max_depth: int = 3, branch_factor: int = 3):
        self.max_depth = max_depth
        self.branch_factor = branch_factor
    
    async def solve(self, problem: str) -> str:
        # Root node
        nodes = [{"path": [], "state": problem, "score": 1.0}]
        
        for depth in range(self.max_depth):
            candidates = []
            for node in nodes:
                # Generate branches
                branches = await self.expand(node["state"])
                for branch in branches[:self.branch_factor]:
                    score = await self.evaluate(branch)
                    candidates.append({
                        "path": node["path"] + [branch],
                        "state": branch,
                        "score": score
                    })
            # Prune to top-k
            nodes = sorted(candidates, key=lambda x: x["score"], reverse=True)[:self.branch_factor]
        
        return nodes[0]["path"][-1]  # Best final state
    
    async def expand(self, state: str) -> list[str]:
        prompt = f"""Current reasoning: {state}
Generate {self.branch_factor} distinct next reasoning steps:"""
        response = await call_model(prompt, temperature=0.8)
        return parse_branches(response)
    
    async def evaluate(self, state: str) -> float:
        prompt = f"""Rate the promise of this reasoning state (0-1): {state}"""
        response = await call_model(prompt, temperature=0)
        return float(extract_number(response))

ToT shines on planning, puzzle solving, and multi-constraint optimization. Cost grows exponentially with depth. Most production systems don’t need full ToT — structured CoT with self-consistency covers 90% of cases.

Integrating with tool use

Chain-of-thought combines naturally with tool calling. The reasoning trace decides which tool to call and why; the tool result feeds back into the next reasoning step.

TOOL_COT = """You have access to these tools:
- calculate(expression: str) -> float
- search(query: str) -> str
- lookup_docs(doc_id: str) -> str

Problem: {problem}

Think step by step. When you need a tool, output:
TOOL: tool_name(args)
Then continue reasoning with the result.

Reasoning:"""

Execution loop:

async def run_tool_cot(problem: str, max_steps: int = 10) -> str:
    conversation = [{"role": "user", "content": TOOL_COT.format(problem=problem)}]
    
    for _ in range(max_steps):
        response = await call_model(conversation)
        conversation.append({"role": "assistant", "content": response})
        
        if "TOOL:" in response:
            tool_call = parse_tool_call(response)
            result = await execute_tool(tool_call)
            conversation.append({"role": "tool", "content": str(result), "tool_call_id": tool_call.id})
        else:
            return extract_final_answer(response)
    
    raise MaxStepsExceeded()

This pattern — reason, act, observe — is the backbone of ReAct-style agents. The chain-of-thought is the agent’s policy.

Common pitfalls

Over-reasoning on simple tasks. Don’t force CoT for classification, extraction, or lookup tasks. It adds latency and tokens for no gain. Route simple queries to direct prompts.

Reasoning leakage. The model sometimes includes the answer inside the reasoning trace, then repeats it. Your parser should extract from the designated answer field, not the last sentence of reasoning.

Hallucinated intermediate values. In math-heavy traces, the model may invent numbers. Mitigation: force tool use for calculation, or verify each step programmatically against a symbolic solver.

Format drift. Even with structured prompts, models occasionally emit markdown fences, extra keys, or commentary. Always wrap parsing in a repair loop (see structured CoT example).

Context window pressure. Few-shot CoT + long problems + tool results = context exhaustion. Truncate old turns, summarize intermediate state, or use a smaller model for the reasoning trace and a larger one for the final answer.

Temperature confusion. Zero-shot CoT often works better at temperature 0 (deterministic). Few-shot and self-consistency need temperature > 0. Document the setting per prompt template.

Evaluation: how to know it’s working

Don’t ship CoT prompts without evals. Build a test set of representative problems with ground-truth answers and expected reasoning patterns.

EVAL_CASES = [
    {
        "problem": "If 3 machines make 3 widgets in 3 minutes, how long for 100 machines to make 100 widgets?",
        "answer": "3 minutes",
        "required_steps": ["rate per machine", "parallel scaling"],
        "forbidden_patterns": ["100 minutes", "300 minutes"]
    },
    # ...
]

def evaluate_cot(prompt_template: str, cases: list[dict]) -> dict:
    results = []
    for case in cases:
        response = call_model(prompt_template.format(problem=case["problem"]))
        reasoning, answer = parse_response(response)
        
        results.append({
            "correct": answer.strip() == case["answer"],
            "has_required_steps": all(s in reasoning for s in case["required_steps"]),
            "has_forbidden": any(f in reasoning for f in case["forbidden_patterns"]),
            "token_count": count_tokens(response)
        })
    
    return {
        "accuracy": sum(r["correct"] for r in results) / len(results),
        "step_coverage": sum(r["has_required_steps"] for r in results) / len(results),
        "hallucination_rate": sum(r["has_forbidden"] for r in results) / len(results),
        "avg_tokens": sum(r["token_count"] for r in results) / len(results)
    }

Track these metrics over prompt iterations. A prompt that gets the right answer but fails step coverage is brittle — it’ll break on distribution shift.

When to use each variant

Start with zero-shot. Measure. Add structure when you need parsing. Add examples when zero-shot fails consistently. Add self-consistency when errors are costly. Escalate complexity only when metrics demand it.

A note on routing

If you’re running multiple models — say, a fast model for simple queries and a reasoning-optimized model for CoT — route at the gateway level. The prompt template stays the same; the model changes. This keeps your application code clean and lets you swap models as new ones arrive. A gateway that honors client routing directives and forwards provider cache-control hints makes this practical without custom infrastructure per provider.

Chain-of-thought isn’t magic. It’s a prompting pattern that exploits how autoregressive models work. Use the simplest variant that passes your evals. Structure the output so your code can trust it. Measure everything.