Chain-of-thought prompting explained with real examples

Chain-of-thought prompting forces a model to emit intermediate reasoning steps before producing a final answer. It works because the tokens generated during reasoning become conditioning context for subsequent tokens — the model literally reads its own thinking. This guide covers the patterns that actually work in production, where they break, and how to implement them without burning tokens on theater.

Zero-shot chain of thought

The simplest form appends a trigger phrase to your prompt. Kojima et al. showed that “Let’s think step by step” alone lifts accuracy on GSM8K from ~18% to ~79% with text-davinci-002. Modern models respond to shorter triggers.

def zero_shot_cot(question: str, model: str = "gpt-4o-mini") -> str:
    prompt = f"{question}\n\nLet's think step by step."
    return call_model(prompt, model=model, max_tokens=512)

This works for arithmetic, symbolic reasoning, and multi-hop QA. It fails when the problem requires domain-specific reasoning patterns the model hasn’t internalized — tax calculations, regex construction, or Kubernetes debug sequences. The model will hallucinate plausible-sounding steps that don’t match your constraints.

Pitfall: Zero-shot CoT often produces verbose, meandering reasoning. Set max_tokens explicitly or you’ll pay for paragraphs of preamble before the answer.

Few-shot chain of thought

Provide 3–8 exemplars showing the exact reasoning format you want. This is where you encode domain logic.

FEW_SHOT_EXAMPLES = """
Q: A server processes 1,200 requests/minute. Each request uses 2 MB RAM. 
   The server has 8 GB RAM. How many minutes until OOM?
A: Step 1: Convert 8 GB to MB → 8 * 1024 = 8192 MB.
   Step 2: RAM per minute → 1200 * 2 = 2400 MB/min.
   Step 3: Minutes until OOM → 8192 / 2400 ≈ 3.41 minutes.
   Answer: 3.41 minutes.

Q: A pod restarts every 45 seconds. CrashLoopBackOff triggers after 5 restarts 
   in 10 minutes. Will it trigger?
A: Step 1: Time for 5 restarts → 5 * 45 = 225 seconds.
   Step 2: 10 minutes = 600 seconds.
   Step 3: 225 < 600, so yes, CrashLoopBackOff triggers.
   Answer: Yes.

Q: {question}
A: Let's solve this step by step.
"""

def few_shot_cot(question: str, model: str = "gpt-4o-mini") -> str:
    prompt = FEW_SHOT_EXAMPLES.format(question=question)
    return call_model(prompt, model=model, max_tokens=512, temperature=0)

Tradeoff: Few-shot CoT costs more tokens per request but dramatically improves adherence to your reasoning schema. For high-volume workloads, distill the few-shot examples into a fine-tuned model or a system prompt with cached exemplars.

Structured chain of thought

Unstructured reasoning is hard to parse. Force a machine-readable format — JSON, XML, or a strict template — so downstream code can extract the answer reliably.

import json
from pydantic import BaseModel, Field
from typing import Literal

class ReasoningStep(BaseModel):
    step: int
    calculation: str | None = None
    result: str | None = None

class CoTResponse(BaseModel):
    reasoning: list[ReasoningStep]
    final_answer: str
    confidence: Literal["high", "medium", "low"]

STRUCTURED_COT_PROMPT = """
You are a precise reasoning engine. Output ONLY valid JSON matching this schema:
{schema}

Question: {question}
"""

def structured_cot(question: str, model: str = "gpt-4o") -> CoTResponse:
    schema = CoTResponse.model_json_schema()
    prompt = STRUCTURED_COT_PROMPT.format(schema=json.dumps(schema), question=question)
    raw = call_model(prompt, model=model, max_tokens=1024, temperature=0, response_format={"type": "json_object"})
    return CoTResponse.model_validate_json(raw)

This eliminates post-processing regexes and lets you validate reasoning completeness programmatically. The schema also acts as a contract — if the model omits calculation fields, your validator catches it before the answer reaches users.

Pitfall: Strict JSON mode increases latency and sometimes degrades reasoning quality on smaller models. Test with gpt-4o-mini vs gpt-4o — the larger model handles constraints better.

Self-consistency decoding

Run the same CoT prompt multiple times at temperature > 0, then take the majority answer. Wang et al. showed this lifts GSM8K accuracy by 10–15% over greedy decoding.

import asyncio
from collections import Counter
from statistics import mode

async def self_consistency(question: str, n: int = 5, model: str = "gpt-4o-mini") -> str:
    prompt = f"{question}\n\nLet's think step by step."
    
    async def single_run():
        return await call_model_async(prompt, model=model, max_tokens=512, temperature=0.7)
    
    responses = await asyncio.gather(*[single_run() for _ in range(n)])
    answers = [extract_final_answer(r) for r in responses]
    
    # Majority vote
    counts = Counter(answers)
    return counts.most_common(1)[0][0]

def extract_final_answer(text: str) -> str:
    # Heuristic: last line after "Answer:" or final numeric value
    for line in reversed(text.strip().split('\n')):
        if line.lower().startswith('answer:'):
            return line.split(':', 1)[1].strip()
    # Fallback: last number-like token
    import re
    numbers = re.findall(r'-?\d+\.?\d*', text)
    return numbers[-1] if numbers else text.strip()[-100:]

Tradeoff: 5x cost and latency for ~10% accuracy gain. Worth it for high-stakes decisions (billing calculations, capacity planning), wasteful for chat. Cache the prompt and run in parallel to bound latency.

Tree of thoughts for search problems

When the reasoning path isn’t linear — debugging, planning, code generation — explore multiple branches and prune.

from dataclasses import dataclass
from typing import Optional

@dataclass
class ThoughtNode:
    state: str           # Current partial solution
    reasoning: str       # How we got here
    score: float         # Heuristic value (0-1)
    parent: Optional['ThoughtNode'] = None
    children: list['ThoughtNode'] = None

async def tree_of_thoughts(
    problem: str, 
    model: str = "gpt-4o",
    breadth: int = 3,
    depth: int = 4,
    threshold: float = 0.3
) -> str:
    root = ThoughtNode(state=problem, reasoning="", score=1.0)
    current_level = [root]
    
    for level in range(depth):
        next_level = []
        for node in current_level:
            # Generate candidate next steps
            candidates = await generate_thoughts(node, model, breadth)
            # Score and filter
            scored = await score_thoughts(candidates, problem, model)
            next_level.extend([c for c in scored if c.score >= threshold])
        
        if not next_level:
            break
        # Keep top-k for next iteration
        current_level = sorted(next_level, key=lambda n: n.score, reverse=True)[:breadth]
    
    # Return best leaf's full reasoning trace
    best = max(current_level, key=lambda n: n.score)
    return reconstruct_path(best)

async def generate_thoughts(node: ThoughtNode, model: str, k: int) -> list[ThoughtNode]:
    prompt = f"""
Problem: {node.state}
Current reasoning: {node.reasoning}

Generate {k} distinct next reasoning steps. Each should be a concrete action or deduction.
Output as JSON array of strings.
"""
    raw = await call_model_async(prompt, model=model, temperature=0.8, response_format={"type": "json_object"})
    steps = json.loads(raw)["steps"]
    return [ThoughtNode(state=node.state, reasoning=node.reasoning + "\n" + s, score=0.0) for s in steps]

async def score_thoughts(nodes: list[ThoughtNode], problem: str, model: str) -> list[ThoughtNode]:
    prompt = f"""
Problem: {problem}
Candidate reasoning paths:
{json.dumps([{"id": i, "reasoning": n.reasoning} for i, n in enumerate(nodes)])}

Score each 0-1 for: progress toward solution, logical validity, relevance.
Output JSON: {{"scores": [{{"id": 0, "score": 0.8}}, ...]}}
"""
    raw = await call_model_async(prompt, model=model, temperature=0, response_format={"type": "json_object"})
    scores = {s["id"]: s["score"] for s in json.loads(raw)["scores"]}
    for i, node in enumerate(nodes):
        node.score = scores.get(i, 0.0)
    return nodes

This is expensive — O(breadth^depth) model calls. Use only when the problem genuinely branches (incident response, architecture decisions). For linear reasoning, self-consistency is cheaper.

Program-aided reasoning

Offload computation to code. The model writes a program, you execute it, and feed the result back. This eliminates arithmetic hallucinations entirely.

import subprocess
import sys
from textwrap import dedent

PROGRAM_AIDED_PROMPT = dedent("""
    Write a Python program to solve this problem. 
    The program must print ONLY the final answer as a JSON object: {{"answer": <value>}}
    No explanations, no markdown, just the JSON.
    
    Problem: {question}
""")

def program_aided_reasoning(question: str, model: str = "gpt-4o") -> str:
    prompt = PROGRAM_AIDED_PROMPT.format(question=question)
    code = call_model(prompt, model=model, max_tokens=1024, temperature=0)
    
    # Extract Python code from markdown if present
    if "```python" in code:
        code = code.split("```python")[1].split("```")[0]
    elif "```" in code:
        code = code.split("```")[1].split("```")[0]
    
    # Execute in isolated subprocess
    result = subprocess.run(
        [sys.executable, "-c", code],
        capture_output=True,
        text=True,
        timeout=10
    )
    
    if result.returncode != 0:
        raise RuntimeError(f"Code execution failed: {result.stderr}")
    
    output = json.loads(result.stdout.strip())
    return str(output["answer"])

Why this works: The model only needs to generate correct logic, not correct arithmetic. Python handles floating point, big integers, and date math perfectly. You also get an audit trail — the code is the reasoning.

Pitfall: Never execute model-generated code in your main process. Use a sandbox (gVisor, Firecracker, or at minimum a separate container with no network/disk access). The example above uses a subprocess for simplicity; production needs stronger isolation.

When not to use chain of thought

Token budgeting

CoT consumes output tokens. A typical GSM8K solution uses 150–300 reasoning tokens. At scale, this matters.

def estimate_cot_cost(questions: list[str], avg_reasoning_tokens: int = 200) -> dict:
    input_tokens = sum(len(q) // 4 for q in questions)  # rough char/4 estimate
    output_tokens = len(questions) * avg_reasoning_tokens
    # gpt-4o-mini pricing (example): $0.15/1M input, $0.60/1M output
    cost = (input_tokens * 0.15 + output_tokens * 0.60) / 1_000_000
    return {
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "estimated_cost_usd": cost
    }

If you’re routing through a gateway that meters per-token usage, you can enforce budgets per request and reject or truncate reasoning that exceeds thresholds.

Debugging failed reasoning

When CoT produces wrong answers, the trace tells you why. Build a debug endpoint that returns the full reasoning alongside the answer.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class CoTRequest(BaseModel):
    question: str
    method: Literal["zero_shot", "few_shot", "structured", "self_consistency"] = "zero_shot"
    return_reasoning: bool = True

class CoTResponse(BaseModel):
    answer: str
    reasoning: str | None = None
    metadata: dict

@app.post("/cot", response_model=CoTResponse)
async def cot_endpoint(req: CoTRequest):
    if req.method == "zero_shot":
        reasoning = await zero_shot_cot_async(req.question)
        answer = extract_final_answer(reasoning)
    elif req.method == "structured":
        result = await structured_cot_async(req.question)
        reasoning = "\n".join(f"Step {s.step}: {s.description} → {s.result}" for s in result.reasoning)
        answer = result.final_answer
    # ... other methods
    
    return CoTResponse(
        answer=answer,
        reasoning=reasoning if req.return_reasoning else None,
        metadata={"method": req.method, "model": "gpt-4o-mini"}
    )

Log every request with its reasoning trace. When accuracy drops, you can diff traces before/after model updates or prompt changes.

Summary checklist

• Start with zero-shot — “Let’s think step by step” costs nothing to try
• Move to few-shot when domain logic matters — 3–5 exemplars usually suffice
• Use structured output for any downstream parsing — JSON schema as contract
• Apply self-consistency only for high-value decisions — 5x cost for ~10% gain
• Offload computation to code — eliminates arithmetic hallucinations
• Budget tokens explicitly — reasoning tokens are output tokens you pay for
• Log reasoning traces — they’re your debug artifacts when things break

Chain-of-thought isn’t magic. It’s a prompting discipline that makes the model’s latent reasoning visible and verifiable. Use the lightest variant that works for your task, measure the accuracy/cost tradeoff, and instrument everything.