In-context learning is the ability of a large language model to perform a new task by conditioning on a few input-output examples provided in the prompt, without any gradient updates to the model’s parameters. The model infers the pattern from the demonstrations and applies it to the query, effectively “learning” the task within the context window. This capability emerges from pre-training on diverse sequences where predicting the next token requires recognizing and adapting to local patterns.
How in-context learning works
At inference time, you construct a prompt containing a task description (optional), several labeled examples called demonstrations, and the target input. The model processes this entire sequence and generates the completion for the target. No backpropagation occurs. The model’s frozen weights implement a function that maps the concatenated context to a prediction.
The mechanism relies on the transformer’s attention layers. Each demonstration provides a (input, output) pair. Attention allows the model to compare the target input against demonstration inputs, retrieve the corresponding outputs, and compose a response consistent with the demonstrated pattern. Research suggests the model implements something akin to gradient descent internally — the forward pass computes an implicit update over the demonstrations — but from the engineer’s perspective, it is a pure inference-time phenomenon.
# Minimal in-context learning prompt structure
prompt = """Classify the sentiment of each review as positive or negative.
Review: "The battery life is incredible, easily lasts two days."
Sentiment: positive
Review: "Screen cracked after one week. Terrible quality control."
Sentiment: negative
Review: "Fast shipping, exactly as described. Will buy again."
Sentiment: positive
Review: "The app crashes constantly and support never responds."
Sentiment:"""
# Model completes with "negative"The number of demonstrations (shots) varies. Zero-shot provides only the task description. Few-shot provides 1–50+ examples. Many-shot can stuff hundreds of examples into long-context windows, often improving performance on tasks with high variance or many classes.
Why it matters for engineers
In-context learning changes the deployment economics of LLMs. Instead of fine-tuning a model per task — which requires GPU compute, labeled datasets, versioned artifacts, and serving infrastructure — you ship a single frozen model and vary the prompt. This reduces the marginal cost of a new task to near zero.
It also enables rapid iteration. You can test prompt variations in seconds, A/B test demonstration selection strategies, and roll back instantly. For teams without ML infrastructure, it is often the only practical path to production.
However, it has limits. Context window consumption scales linearly with demonstrations. Each example costs input tokens at inference time, increasing latency and cost. For high-volume tasks, fine-tuning a smaller model often wins on total cost of ownership. In-context learning also degrades when demonstrations are noisy, contradictory, or poorly ordered. The model has no mechanism to “unlearn” a bad example once it sits in the context.
Concrete example: structured extraction
Consider extracting structured fields from unstructured emails. A zero-shot prompt might work for simple cases but fails on edge cases — missing fields, ambiguous phrasing, multiple entities. A few-shot prompt with curated demonstrations handles these reliably.
{
"task": "Extract order details from customer emails",
"demonstrations": [
{
"input": "Hi, I ordered the Pro X headphones (SKU: HX-900) on March 15 but received the wrong color. Order #ORD-8821. Please exchange for matte black.",
"output": {
"order_id": "ORD-8821",
"product_sku": "HX-900",
"issue": "wrong_color",
"requested_action": "exchange",
"preferred_variant": "matte_black"
}
},
{
"input": "My order ORD-9102 for the standing desk (DS-200) arrived with a damaged tabletop. I'd like a refund instead of replacement.",
"output": {
"order_id": "ORD-9102",
"product_sku": "DS-200",
"issue": "damaged",
"requested_action": "refund",
"preferred_variant": null
}
}
],
"query": "Order ORD-7741: wireless charger WC-100. The LED indicator doesn't light up. Can you send a replacement?",
"expected_output": {
"order_id": "ORD-7741",
"product_sku": "WC-100",
"issue": "defective",
"requested_action": "replacement",
"preferred_variant": null
}
}With two demonstrations, the model learns the output schema, the field names, the enum values for issue and requested_action, and the null convention for preferred_variant. Adding a third demonstration covering a cancellation request would extend coverage without retraining.
Demonstration design principles
The quality of in-context learning depends almost entirely on demonstration selection and ordering. Random examples from a dataset often underperform curated ones.
Diversity beats similarity. Demonstrations should cover the space of valid inputs, not cluster around the query. If your query is a refund request, include exchanges, cancellations, and warranty claims — not five refund examples. Coverage reduces the chance the model overfits to a narrow pattern.
Order matters. Recency bias is real: demonstrations near the query exert disproportionate influence. Place the most representative or difficult examples last. Some practitioners sort by semantic similarity to the query (retrieval-augmented in-context learning), but this can create echo chambers. A deterministic order — e.g., by difficulty or class balance — is often more robust.
Format consistency is non-negotiable. Every demonstration must follow the exact same schema, whitespace, and punctuation. The model learns the format as strongly as the task. A single demonstration with a trailing comma or missing field can corrupt outputs across the batch.
Label quality exceeds quantity. Ten clean, diverse demonstrations beat fifty noisy ones. If you have limited annotation budget, spend it on verification, not volume.
Common misconceptions
“In-context learning is just pattern matching”
Pattern matching implies shallow n-gram overlap. In-context learning exhibits compositional generalization: the model combines concepts from demonstrations in novel ways. Give it examples of “add 2” and “multiply by 3” on numbers, then ask for “add 2 then multiply by 3” on a new number — it often succeeds. This suggests the model learns an algorithm, not a lookup table.
“More shots always help”
Performance saturates, then sometimes degrades. Irrelevant demonstrations add noise and consume context. For classification with k classes, k to 2k diverse examples often suffice. Beyond that, marginal gains diminish while token costs grow linearly. Many-shot (hundreds of examples) helps only for tasks with long-tailed distributions or where the model must memorize specific facts.
“It replaces fine-tuning”
It replaces fine-tuning for prototyping and low-volume tasks. For high-throughput, latency-sensitive, or highly specialized domains (legal coding, medical coding, proprietary DSLs), a fine-tuned 7B or 13B model outperforms a prompted 70B+ model at lower cost. In-context learning is a capability, not a universal substitute.
“The model understands the task”
The model does not “understand” in any semantic sense. It predicts tokens consistent with the conditional distribution defined by the prompt. Adversarial demonstrations — examples with flipped labels, contradictory instructions, or injected personas — can hijack the output. Treat the context as untrusted input if demonstrations come from external sources.
“Temperature zero guarantees determinism”
Even at temperature 0, different hardware, batching, or kernel implementations can produce different tokens due to floating-point non-determinism in attention softmax. For reproducible outputs, you need fixed seeds, deterministic kernels, and identical hardware — or you accept stochasticity and design for it.
Evaluation strategy
You cannot ship in-context learning without evaluation. Build a held-out test set representative of production traffic. Measure exact match, F1, or task-specific metrics across multiple random seeds and demonstration orderings. Track:
- Variance across seeds: High variance means your prompt is brittle.
- Variance across demonstration subsets: If swapping two examples changes accuracy by >5%, your demonstration set is underspecified.
- Failure mode taxonomy: Categorize errors (schema violation, wrong label, hallucinated field) to guide demonstration curation.
Automate this. A CI job that runs your prompt against the test set on every prompt change catches regressions before they hit production.
When to reach for fine-tuning instead
Switch to fine-tuning when:
- Inference token cost exceeds fine-tuning amortized cost at your volume
- Latency budget excludes the demonstration tokens (e.g., <100ms p99)
- The task requires knowledge not present in pre-training (proprietary schemas, internal codes)
- You need guaranteed schema adherence — fine-tuned models can be constrained via grammar-based decoding more reliably than prompted models
- Regulatory or compliance requirements demand model auditability and fixed weights
A pragmatic pipeline: start with in-context learning, collect production logs, curate a high-quality dataset from corrected outputs, then fine-tune a smaller model for the hot path. Keep the prompted large model as a fallback for long-tail inputs.
Tooling note
If you route requests across multiple model providers, demonstration formatting must survive provider-specific chat templates. Some providers inject system prompts, strip whitespace, or apply non-standard tokenization to the final assistant message. Test your prompt on each target model. A gateway that normalizes chat templates and forwards provider cache-control hints can reduce the integration surface — n4n.ai handles this by presenting one OpenAI-compatible endpoint across 240+ models while preserving provider-specific behaviors like prompt caching.
Summary
In-context learning is a real, useful capability: frozen models adapt to new tasks from demonstrations in the prompt. It works because transformers implement an implicit learning algorithm over the context window. For engineers, it enables zero-deployment-cost task adaptation at the price of inference tokens and context length. Design demonstrations for diversity, consistency, and coverage. Evaluate rigorously. Graduate to fine-tuning when volume or latency demands it. The prompt is your model artifact — version it, test it, and treat it like code.