What is a reasoning model?

A reasoning model is an LLM trained to generate explicit intermediate reasoning steps — often called chain-of-thought — before producing a final answer, and optimized to use additional test-time compute to improve accuracy on complex tasks. Unlike standard instruction-tuned models that predict the next token conditioned only on the prompt, reasoning models are trained with reinforcement learning to produce verifiable reasoning traces that lead to correct solutions. This architectural shift moves computation from training-time (bigger models, more data) to inference-time (more tokens generated per query).

How reasoning models work

The core mechanism is straightforward: the model generates a sequence of reasoning tokens before emitting the final answer. During training, this behavior is shaped through a combination of supervised fine-tuning on high-quality reasoning traces and reinforcement learning with verifiable rewards — typically correctness on math, code, or logic benchmarks.

# Conceptual training loop for a reasoning model
def train_reasoning_model(base_model, reasoning_dataset, verifier):
    # Phase 1: Supervised fine-tuning on curated reasoning traces
    model = sft(base_model, reasoning_dataset)
    
    # Phase 2: RL with outcome-based rewards
    for batch in dataloader:
        # Generate multiple reasoning traces per problem
        traces = model.generate(batch.prompts, num_return_sequences=8)
        
        # Verify each trace (exact match, unit tests, formal proof)
        rewards = [verifier(trace, batch.ground_truth) for trace in traces]
        
        # Policy gradient update favoring correct traces
        loss = -logprob(traces) * rewards
        loss.backward()
        optimizer.step()

At inference time, the model produces reasoning tokens sequentially. The key insight: more tokens = more compute = better answers, up to a point. This is test-time compute scaling. You can control it explicitly:

# OpenAI-compatible API call with reasoning control
response = client.chat.completions.create(
    model="o1-preview",  # or deepseek-r1, qwen-qwq, etc.
    messages=[{"role": "user", "content": "Solve: ∫₀¹ x² ln(x) dx"}],
    max_completion_tokens=8192,  # budget for reasoning + answer
    # Some providers expose reasoning_effort: "low" | "medium" | "high"
)

The reasoning trace is not decorative. It serves three functions: (1) it decomposes the problem into tractable substeps, (2) it enables self-correction within the same generation, and (3) it provides an audit trail for verification. Models like DeepSeek-R1, OpenAI o1, and Qwen-QwQ emit these traces as part of the completion stream — you see the thinking in real time.

Why it matters for engineers

Reasoning models change the cost/accuracy tradeoff fundamentally. With standard LLMs, you improve accuracy by: bigger model, better prompt, RAG, fine-tuning, or ensembling. With reasoning models, you have a new knob: inference tokens.

This means you can take a single deployed model and dial accuracy per-request. A coding agent might use 2k tokens for a simple refactor and 32k for a subtle concurrency bug. The model is the same; the compute budget differs.

Practical implications:

• Latency budgets become accuracy budgets. If your p99 is 30s, that’s your token budget. Plan accordingly.
• Streaming is non-negotiable. Users need to see progress during 10-60s generations. Implement token-by-token streaming with partial rendering.
• Token costs dominate. At $15-60/M output tokens (current frontier pricing), a 32k reasoning trace costs $0.50-2.00 per query. Build token budgets into your product economics.
• Caching gets harder. Reasoning traces are rarely identical. Standard prefix caching helps only on the prompt side.

Concrete example: debugging a race condition

Here’s a real scenario where a reasoning model earns its keep. Standard GPT-4o misses the bug; a reasoning model catches it.

Prompt:

# Find the bug in this async cache implementation
class AsyncCache:
    def __init__(self):
        self._cache = {}
        self._locks = {}
    
    async def get_or_set(self, key, factory):
        if key not in self._cache:
            if key not in self._locks:
                self._locks[key] = asyncio.Lock()
            async with self._locks[key]:
                # Double-checked locking
                if key not in self._cache:
                    self._cache[key] = await factory()
        return self._cache[key]

Standard model response (abridged):

The code looks correct. It uses double-checked locking with per-key locks to avoid redundant factory calls. The async with ensures proper lock release.

Reasoning model trace (condensed):

Let me analyze this step by step.

1. First check: `if key not in self._cache:` - no lock held, race window opens
2. Lock creation: `if key not in self._locks:` - ANOTHER race window! Two coroutines 
   can create different Lock objects for the same key
3. First coroutine acquires lock A, second acquires lock B - they're different locks!
4. Both enter the critical section, both call factory(), both write to cache
5. Factory runs twice. If factory has side effects (API call, DB write), they happen twice.

The bug: lock creation is not atomic. Fix: use a single global lock for lock creation,
or use `setdefault` on the locks dict, or use a concurrent dict.

Corrected version:
    lock = self._locks.setdefault(key, asyncio.Lock())
    async with lock:
        ...

The reasoning model didn’t just know the answer — it worked through the interleaving. That’s the difference.

Common misconceptions

“Reasoning models are just chain-of-thought prompting”

False. Prompting a base model with “think step by step” produces performative reasoning — the model mimics the style without the underlying capability. Reasoning models are trained to reason. The RL phase shapes the policy to generate traces that actually lead to correct answers, not just plausible-sounding ones. You can see the difference on benchmarks: prompted CoT plateaus; trained reasoning scales with tokens.

“They’re only for math and code”

Math and code are the easiest domains to verify during RL (unit tests, exact match, formal verification), so they dominate benchmarks. But the capability transfers. Any problem that benefits from decomposition — legal analysis, medical differential diagnosis, architectural review, threat modeling — sees gains. The model learns a general heuristic: break it down, check each step, correct course.

“More reasoning tokens always help”

Diminishing returns hit hard. On AIME 2024, going from 4k to 16k tokens might jump accuracy 15 points; 16k to 64k might add 2. Beyond the “sufficient reasoning” threshold, the model hallucinates or loops. Each model family has a sweet spot. Profile your workloads.

“You need the biggest reasoning model”

Smaller distilled reasoning models (DeepSeek-R1-Distill-Qwen-14B, 32B) often match larger ones on domain-specific tasks if you give them adequate token budgets. A 14B model with 32k tokens can beat a 70B model with 4k tokens on structured reasoning. Test your actual workload before defaulting to the largest model.

“Reasoning traces are human-readable by design”

They’re readable enough for debugging, but not optimized for human consumption. Models use shorthand, skip “obvious” steps, and occasionally switch languages mid-trace. Don’t build UX that assumes clean pedagogical explanations. Build UX that shows raw tokens with collapse/expand and search.

Operational considerations

Routing: Not every query needs reasoning. Classify incoming requests — simple lookup, classification, extraction go to fast/cheap models. Reserve reasoning models for multi-step problems. This is where a gateway that supports per-request model selection pays off.

Timeouts: Set generation timeouts per use case. A chat assistant might tolerate 60s; an autocomplete must return in 500ms. Reasoning models are the wrong tool for latency-critical paths.

Evaluation: Standard evals (MMLU, GSM8K) don’t capture reasoning quality on your domain. Build a golden set of 50-100 representative problems with verified solutions. Track pass@k with varying token budgets. This is your real benchmark.

Observability: Log reasoning token count, latency, and outcome separately from answer tokens. You’ll need this data to tune budgets and detect regressions when model versions change.

{
  "request_id": "req_abc123",
  "model": "deepseek-r1",
  "prompt_tokens": 1240,
  "reasoning_tokens": 8432,
  "answer_tokens": 512,
  "total_latency_ms": 14200,
  "outcome": "correct",
  "token_budget": 16384
}

When not to use them

• Single-hop QA with clear answers in context
• High-throughput classification or extraction
• Real-time user-facing latency < 2s
• Domains where reasoning provides no decomposition benefit (translation, style transfer, simple summarization)
• When you can’t afford the token variance — reasoning traces range from 500 to 50k tokens for similar-looking prompts

The bottom line

Reasoning models are a new primitive: programmable test-time compute. You trade latency and token cost for accuracy on a per-request basis. That’s a powerful lever, but it demands new operational muscles — token budgeting, streaming UX, domain-specific evals, and smart routing. Teams that treat them as “just a slower, smarter model” will overspend and underdeliver. Teams that build around the compute/accuracy curve will solve problems that were impractical six months ago.