Few-shot prompting for classification tasks

Few-shot prompting classification works because the model learns the task structure from demonstrations rather than instructions alone. You give the model a handful of labeled examples in the prompt, and it generalizes the pattern to new inputs. This approach consistently outperforms zero-shot for nuanced categories, ambiguous boundaries, and domain-specific taxonomies — provided you select the right examples and format them cleanly.

When few-shot beats zero-shot

Zero-shot works when your categories are semantically distinct and well-represented in the model’s training data: “spam vs not spam,” “positive vs negative sentiment,” “English vs Spanish.” The model already knows these boundaries.

Reach for few-shot when:

• Categories are similar or hierarchical (e.g., “billing issue” vs “account issue” vs “technical issue”)
• Domain jargon shifts meaning (“ticket” means something different in IT support vs event management)
• You need consistent output formatting (JSON, specific labels, confidence scores)
• Edge cases matter and you can show the model how to handle them

The tradeoff is token cost and latency. Each example consumes context window and increases per-request tokens. For high-volume classification, this adds up. A 10-example prompt with 200-token inputs costs ~2,000 extra tokens per request compared to zero-shot.

Selecting examples that teach

Random examples hurt more than they help. The model learns from which examples you show, not just that you showed examples.

Prioritize boundary cases. If “refund request” and “billing inquiry” are your two hardest categories to distinguish, include 2-3 examples of each where the distinction is subtle but the label is clear. The model learns the decision boundary from these.

Cover label distribution. If production traffic is 70% “general inquiry,” 20% “technical issue,” 10% “billing,” your few-shot examples should roughly mirror this. Over-representing rare classes teaches the model a distorted prior.

Include negative examples for exclusionary categories. If you have an “other” or “out-of-scope” bucket, show the model what doesn’t belong in your defined categories:

{
  "input": "What's the weather in Tokyo?",
  "label": "out_of_scope",
  "reasoning": "Weather queries are not covered by our support taxonomy"
}

Limit to 5-8 examples per class. Beyond this, marginal returns drop and token costs rise. For 5+ classes, consider 3-4 examples per class rather than 8 for 2 classes.

Formatting for parseability

The model needs to recognize the pattern instantly. Inconsistent formatting forces the model to waste capacity on parsing instead of classifying.

Use a consistent template with clear delimiters:

### Example 1
Input: "I've been charged twice for my subscription"
Label: billing_issue
Confidence: 0.95

### Example 2
Input: "How do I reset my password?"
Label: account_access
Confidence: 0.98

For structured output, demonstrate the exact format you want back:

### Example 1
Input: "My API key stopped working"
Output: {"label": "technical_issue", "confidence": 0.92, "subcategory": "authentication"}

Avoid prose explanations in the few-shot block. They consume tokens and introduce noise. If reasoning helps, add a separate “reasoning” field in your structured output — but keep the few-shot examples minimal.

Example ordering matters

Models exhibit recency bias: examples near the end of the prompt influence predictions more than those at the start. Use this deliberately.

Order by difficulty. Place the hardest, most ambiguous examples last. The model’s final “impression” before seeing the target input should be the nuanced decision boundaries.

Group by class, then shuffle within groups. Don’t alternate classes (A, B, A, B). Present all class A examples, then all class B. This lets the model form a coherent prototype per class before switching contexts.

For many classes, use a “progressive” ordering. Start with the most frequent/easiest classes, progress to rare/hard ones. This mirrors the natural distribution the model will encounter.

def build_few_shot_prompt(examples_by_class: dict[str, list[Example]], 
                          target_input: str) -> str:
    """Build prompt with deliberate ordering."""
    # Sort classes by frequency (descending)
    sorted_classes = sorted(examples_by_class.keys(), 
                           key=lambda c: -class_frequency[c])
    
    prompt_parts = ["Classify the following input:\n"]
    
    for cls in sorted_classes:
        # Take up to 3 examples per class, hardest last
        selected = select_representative_examples(examples_by_class[cls], k=3)
        for ex in selected:
            prompt_parts.append(format_example(ex))
    
    prompt_parts.append(f"\nInput: {target_input}\nOutput:")
    return "\n".join(prompt_parts)

Token budget and context management

Every token in your few-shot block is a token you can’t use for the input or output. At scale, this is real money.

Compress examples aggressively. Strip whitespace, use abbreviations the model understands, remove redundant words:

# Verbose (180 tokens)
Input: "The customer is reporting that they have been billed twice for the same subscription period and they want a refund for the duplicate charge."
Label: billing_issue

# Compressed (45 tokens)
Input: "Billed twice for same subscription, needs refund"
Label: billing_issue

The compressed version works equally well for classification. The model doesn’t need natural language fluency in examples — it needs signal.

Use dynamic example selection. Don’t stuff the same 20 examples into every request. Retrieve the most relevant 5-8 examples for the specific input using embeddings or keyword matching:

def select_few_shot_examples(input_text: str, 
                            example_pool: list[Example], 
                            k: int = 6) -> list[Example]:
    """Select k most relevant examples via embedding similarity."""
    input_emb = embed(input_text)
    scored = [(ex, cosine_sim(input_emb, ex.embedding)) 
              for ex in example_pool]
    scored.sort(key=lambda x: -x[1])
    return [ex for ex, _ in scored[:k]]

This keeps prompts small while maintaining relevance. The tradeoff: an extra embedding lookup per request (negligible latency) and maintaining an example embedding index.

Consider prompt caching. If you use the same few-shot block across many requests, providers like Anthropic and OpenAI offer prompt caching discounts. Structure your prompt so the few-shot block is a stable prefix:

[SYSTEM PROMPT]
[FEW-SHOT EXAMPLES - stable across requests]
[USER INPUT - varies per request]

Common pitfalls

Leakage from example ordering. If you always put “billing_issue” examples before “technical_issue,” the model develops a positional bias. Shuffle class order across prompt versions or use the progressive ordering above.

Overfitting to example phrasing. If all your “refund request” examples contain the word “refund,” the model learns “refund” → “refund_request” rather than the actual semantic boundary. Diversify phrasing: “money back,” “charge reversal,” “dispute charge,” “return my payment.”

Ignoring confidence calibration. Few-shot prompts often produce overconfident predictions. The model sees clean examples and mimics the certainty. Add a calibration step:

def calibrate_confidence(logprobs: list[float], 
                         temperature: float = 0.0) -> float:
    """Convert logprobs to calibrated confidence."""
    if temperature == 0.0:
        # Greedy decoding - use margin between top-2
        top1, top2 = logprobs[0], logprobs[1] if len(logprobs) > 1 else -10
        margin = top1 - top2
        return 1 / (1 + math.exp(-margin * 2))  # sigmoid scaling
    # For sampled outputs, use entropy-based calibration
    probs = [math.exp(lp) for lp in logprobs]
    entropy = -sum(p * math.log(p) for p in probs if p > 0)
    return 1 - (entropy / math.log(len(probs)))

Single-turn evaluation. Testing few-shot prompts on a static test set misses distribution shift. Evaluate on production-like data: recent logs, adversarial examples, out-of-distribution inputs. Track per-class F1, not just accuracy.

Evaluation and iteration loop

Treat few-shot prompting as a machine learning workflow, not a one-off prompt engineering task.

Build a labeled evaluation set. Start with 200-500 examples covering your production distribution. Include known-hard cases. Version this dataset.

Measure per-class metrics. Macro F1 matters more than accuracy when classes are imbalanced. Track confusion matrices to see which pairs the model conflates.

Iterate on examples, not instructions. When the model confuses class A and B, add a clarifying example pair showing the distinction. Don’t add “pay attention to the difference between A and B” to the system prompt — it rarely works.

A/B test example sets. Run two example selections against live traffic (shadow mode or small percentage). Compare latency, token cost, and classification quality. Promote the winner.

Automate regression testing. When you update the example pool, run the full eval suite. CI should fail if macro F1 drops >1% or any class F1 drops >3%.

Production considerations

Latency budgets. Few-shot adds 500-2000 tokens to the prompt. At 50 tokens/ms (typical for 70B models), that’s 10-40ms extra latency per request. For p99 < 200ms budgets, this matters. Profile end-to-end.

Fallback strategy. When the primary model is degraded, your few-shot prompt may not transfer cleanly to a smaller fallback model. Maintain a zero-shot variant for fallback, or a compressed few-shot version (3 examples instead of 8).

Monitoring label drift. Track the distribution of predicted labels over time. A sudden shift often indicates upstream data changes, not model degradation. Alert on KL divergence from baseline distribution.

Example freshness. Retire examples older than 90 days unless they represent timeless boundary cases. Language evolves, product features change, and old examples teach obsolete patterns.

Structured output enforcement. Use constrained decoding or JSON schema validation on the output. Few-shot examples demonstrate the format; the parser enforces it. Never trust the model to output valid JSON 100% of the time.

from pydantic import BaseModel, Field
from typing import Literal

class ClassificationOutput(BaseModel):
    label: Literal["billing_issue", "technical_issue", "account_access", 
                   "general_inquiry", "out_of_scope"]
    confidence: float = Field(ge=0.0, le=1.0)
    subcategory: str | None = None
    reasoning: str | None = None

def parse_classification(raw_output: str) -> ClassificationOutput:
    """Parse with validation, fallback to safe default."""
    try:
        data = json.loads(raw_output)
        return ClassificationOutput(**data)
    except (json.JSONDecodeError, ValidationError):
        # Log for review, return safe fallback
        logger.warning(f"Parse failed: {raw_output[:200]}")
        return ClassificationOutput(
            label="general_inquiry",
            confidence=0.5,
            reasoning="Parse failure - defaulted"
        )

Scaling beyond few-shot

Few-shot prompting classification hits a ceiling. When you need:

• 95% accuracy on nuanced categories

• Sub-100ms latency at scale
• Frequent taxonomy updates without prompt rewrites
• Explainable decisions for compliance

…it’s time to distill the few-shot prompt into a fine-tuned classifier or a smaller dedicated model. The few-shot prompt becomes your training data generator: run it over millions of unlabeled inputs, review high-confidence predictions, and build a supervised dataset.

The prompt that taught the model becomes the teacher that labels the data. That’s the natural progression — not a failure of prompting, but its graduation.