GSM8K (Grade School Math 8K) is a benchmark of 8,500 linguistically diverse grade-school math word problems that require multi-step reasoning to solve. Created by OpenAI in 2021, it tests whether language models can chain arithmetic operations — addition, subtraction, multiplication, division — in the correct order rather than simply memorizing answers. The dataset has become the standard reference for measuring basic mathematical reasoning in LLMs, though its ceiling is now saturated by frontier models.
How the dataset is constructed
Each problem in GSM8K is a short natural-language scenario: “Natalia sold 48 clips in April. She sold half as many in May. How many clips did she sell altogether?” The solution requires two steps — divide 48 by 2, then add the result to 48 — and the dataset provides both the final answer (72) and a step-by-step natural-language rationale.
The 8,500 problems split 7,500 train / 1,000 test. Problems are annotated with the number of reasoning steps required (2–8 steps, median 4). Crucially, the training set includes the rationales, enabling supervised fine-tuning on chain-of-thought traces. The test set withholds rationales; evaluation checks only the final numeric answer.
{
"question": "A robe takes 2 bolts of blue fiber and 3 bolts of white fiber. How many bolts of fiber are needed for 5 robes?",
"answer": "25",
"rationale": "Each robe needs 2 + 3 = 5 bolts. For 5 robes: 5 * 5 = 25 bolts."
}The vocabulary is intentionally simple — mostly one- and two-syllable words — so reading comprehension is not the bottleneck. The difficulty comes entirely from composing arithmetic operations in the right sequence.
Why multi-step reasoning matters
Single-step arithmetic (“What is 48 + 24?”) is trivial for any model that has seen multiplication tables during pretraining. GSM8K’s value is forcing the model to plan: identify sub-problems, execute them in dependency order, and carry intermediate results forward without losing track.
This mirrors how humans solve unfamiliar problems. A model that scores 90% on GSM8K demonstrates it can maintain a small working memory across 4–8 serial operations — a prerequisite for more complex tasks like code generation, financial modeling, or multi-hop QA.
The benchmark also exposes a specific failure mode: models that pattern-match keywords (“half as many” → divide by 2) but fail when the linguistic structure deviates from training distribution. For example, “She sold 48 clips in April. In May she sold 24 fewer clips than in April” requires subtraction, not division, though both contain “fewer.”
Evaluation methodology
Standard evaluation uses exact-match accuracy on the final numeric answer. The answer is normalized (commas removed, trailing zeros trimmed) before comparison. No partial credit for correct reasoning with arithmetic errors.
def normalize_answer(text: str) -> str:
"""Extract and normalize the final numeric answer."""
# Find the last number in the response
numbers = re.findall(r'-?\d[\d,]*\.?\d*', text.replace(',', ''))
if not numbers:
return ""
return numbers[-1].rstrip('.').lstrip('0') or '0'
def gsm8k_accuracy(predictions: list[str], references: list[str]) -> float:
correct = sum(
normalize_answer(p) == normalize_answer(r)
for p, r in zip(predictions, references)
)
return correct / len(predictions)Chain-of-thought prompting is standard. A typical few-shot prompt includes 8 exemplars with full rationales, then the test question. Models that generate their own rationale before the final answer consistently outperform direct-answer prompting by 15–25 percentage points.
Representative results (as of 2024)
| Model | Test accuracy (CoT) |
|---|---|
| GPT-3.5-turbo | ~77% |
| GPT-4 | ~92% |
| Claude 3 Opus | ~95% |
| Llama-3-70B-Instruct | ~93% |
| Phi-3-mini-4k | ~85% |
Frontier models now saturate the benchmark. The remaining errors cluster in three categories: (1) problems requiring 7+ steps where the model loses track, (2) linguistic variations not well-represented in training (e.g., “times fewer” constructions), and (3) unit conversions embedded in the word problem (feet to inches, dollars to cents).
A concrete worked example
Problem: “James has 3 gallons of paint. He uses 1.5 gallons for the living room and 0.75 gallons for the bedroom. How many quarts of paint remain?”
Step-by-step reasoning:
- Total used = 1.5 + 0.75 = 2.25 gallons
- Remaining gallons = 3 - 2.25 = 0.75 gallons
- Convert to quarts: 0.75 × 4 = 3 quarts
Answer: 3
A model that answers “0.75” missed the unit conversion. A model that answers “3 gallons” missed the question’s request for quarts. Both errors appear frequently in sub-90% models.
Common misconceptions
“GSM8K measures math ability”
It measures multi-step arithmetic reasoning expressed in natural language. It does not test algebra, geometry, calculus, or symbolic manipulation. A model can ace GSM8K and fail completely on MATH (the Hendrycks dataset with competition-level problems) or on simple variable isolation like “Solve for x: 3x + 7 = 22.”
“Training on GSM8K train set is cheating”
Using the 7,500 training problems for supervised fine-tuning is standard practice and explicitly intended by the dataset authors. The test set remains held out. What is cheating: training on the test set, or using external tools (calculator, code interpreter) during evaluation unless the benchmark explicitly permits it.
“High GSM8K score means the model is good at reasoning”
It means the model is good at this specific flavor of reasoning: short-horizon, discrete arithmetic with explicit numbers. It does not generalize to:
- Long-horizon planning (10+ steps)
- Reasoning with unknowns or variables
- Spatial or geometric reasoning
- Logical deduction without arithmetic
“The rationales are gold-standard”
The provided rationales are human-written but not verified for minimality or correctness in every case. Some contain redundant steps; a few have subtle arithmetic errors that propagated into the dataset. When fine-tuning on rationales, expect to inherit those quirks.
Where GSM8K fits in an evaluation suite
GSM8K is a necessary but insufficient checkpoint. A reasonable evaluation ladder for mathematical reasoning:
- GSM8K — basic multi-step arithmetic, natural language
- SVAMP — same difficulty, more linguistic variation, tests robustness
- MATH — algebra, geometry, calculus, competition problems
- GPQA — graduate-level physics/chemistry/biology reasoning
- Custom domain evals — your actual use-case problems (pricing logic, dosage calc, etc.)
If you’re building a tutoring agent, GSM8K is a reasonable proxy for “can it walk a 5th grader through a word problem.” If you’re building a financial analyst, GSM8K tells you almost nothing — you need MATH-adjacent evals with variables, units, and multi-table lookups.
Practical tips for engineers
Prompt formatting matters. The canonical few-shot prompt uses #### as a delimiter between rationale and final answer. Match this format exactly when evaluating; deviating can drop scores 3–5 points.
Q: Natalia sold 48 clips in April. She sold half as many in May. How many clips did she sell altogether?
A: Natalia sold 48/2 = 24 clips in May. So altogether she sold 48 + 24 = 72 clips.
#### 72Temperature 0 is not always best. For GSM8K, temperature 0.3–0.7 with self-consistency (sample 8–32 traces, take majority vote) often beats greedy decoding by 2–4 points, especially on 6+ step problems.
Watch for tokenization artifacts. Numbers like “1,000” tokenize differently than “1000.” If your tokenizer splits the comma, the model may treat them as separate tokens and mis-add. Normalize inputs to plain integers where possible.
Don’t over-optimize for GSM8K. Prompt engineering that gains 3 points on GSM8K often hurts performance on out-of-distribution math. Optimize for your actual task distribution.
The saturation problem
GSM8K is effectively solved. The gap between the best open model and the best closed model is <3%. New models routinely hit 94–96% on first release. The benchmark no longer discriminates at the frontier.
This creates a measurement problem: you cannot use GSM8K to compare two 2024-era frontier models. You need harder benchmarks (MATH, GPQA, TheoremQA) or, better, a custom eval that mirrors your production workload.
If you’re running evals through a gateway that routes across 240+ models, GSM8K is still useful as a smoke test — if a model scores below 70%, something is fundamentally broken with its reasoning or the prompt format. But for model selection among competent models, move up the ladder.
Closing thought
GSM8K earned its place as the “hello world” of mathematical reasoning evaluation. It isolates a clean capability — chaining arithmetic in language — and measures it reliably. But like any hello world, it proves the runtime works, not that the application is correct. Treat it as a gate, not a goal.