How much data do you need to fine-tune a model

The question of how much data to fine-tune a model has a frustratingly honest answer: it depends on the task, the base model, and your quality bar. But engineers need starting points, not philosophy. For most instruction-tuning or domain-adaptation tasks on modern 7B–70B parameter models, you can see measurable improvement with 500–2,000 high-quality examples, while full domain mastery or complex reasoning often demands 10,000–100,000. Below that range you’re usually better off with prompt engineering or RAG; above it, diminishing returns hit hard unless you’re training from scratch.

Start with the rule of thumb

A widely cited heuristic from the LLaMA and Alpaca papers: 1,000–2,000 diverse, high-quality instruction-response pairs beats 50,000 noisy ones. This holds for supervised fine-tuning (SFT) on general chat or instruction following. For specialized tasks — code generation, SQL, legal review, medical summarization — the floor rises to 5,000–20,000 examples if you want the model to reliably outperform a strong few-shot prompt.

# Rough data estimation for SFT
def estimate_sft_examples(task_complexity: str, model_size_b: int) -> tuple[int, int]:
    """
    Returns (minimum, recommended) example counts for instruction tuning.
    """
    base = {
        "simple": 500,      # classification, sentiment, formatting
        "moderate": 2000,   # summarization, QA, style transfer
        "complex": 10000,   # code, reasoning, multi-step, domain expertise
    }[task_complexity]

    # Larger models need more data to move the needle, but also generalize better
    scale_factor = max(1.0, model_size_b / 7.0) ** 0.5
    return int(base * scale_factor), int(base * scale_factor * 3)

# Examples:
# estimate_sft_examples("moderate", 7)   -> (2000, 6000)
# estimate_sft_examples("complex", 70)   -> (38000, 114000)

These numbers assume clean, deduplicated, correctly formatted data. If your dataset has label noise, formatting inconsistencies, or near-duplicates, multiply by 3–5x.

What actually determines data requirements

Task type and output space

Base model capability

A stronger base model (e.g., Llama-3-70B-Instruct vs. Llama-2-7B) requires less data to adapt because it already possesses the underlying reasoning and language skills. You’re teaching specialization, not competence. Conversely, a weak base model needs more examples to overcome its priors.

# Adjusting for base model strength
BASE_MODEL_FACTORS = {
    "llama-3-70b-instruct": 0.5,
    "llama-3-8b-instruct": 0.8,
    "mistral-7b-instruct-v0.3": 0.9,
    "llama-2-7b-chat": 1.2,
    "llama-2-13b-chat": 1.0,
    "phi-3-mini-4k-instruct": 1.1,
    "custom-pretrained-7b": 1.5,  # no instruction tuning
}

def adjusted_minimum(base_min: int, base_model: str) -> int:
    factor = BASE_MODEL_FACTORS.get(base_model.lower(), 1.0)
    return int(base_min * factor)

Data quality dimensions

Quality trumps quantity. A 2,000-example dataset that scores high on these dimensions outperforms 20,000 scraped examples:

1. Instruction diversity — Coverage of edge cases, phrasing variations, difficulty levels
2. Response correctness — Factual accuracy, no hallucinations, follows constraints
3. Formatting consistency — Same template, same delimiters, no mixed styles
4. Deduplication — Near-duplicates (embedding cosine > 0.9) waste compute and cause overfitting
5. Distribution match — Training distribution mirrors inference distribution

# Quick quality audit
from datasets import load_dataset
import numpy as np
from sentence_transformers import SentenceTransformer

def audit_dataset(dataset_path: str, sample_size: int = 1000) -> dict:
    ds = load_dataset("json", data_files=dataset_path, split="train").shuffle(seed=42).select(range(sample_size))
    
    # 1. Length stats
    instr_lens = [len(ex["instruction"].split()) for ex in ds]
    resp_lens = [len(ex["output"].split()) for ex in ds]
    
    # 2. Near-duplicate detection (sample)
    embedder = SentenceTransformer("all-MiniLM-L6-v2")
    texts = [ex["instruction"] + " " + ex["output"] for ex in ds]
    embs = embedder.encode(texts, normalize_embeddings=True)
    sims = embs @ embs.T
    np.fill_diagonal(sims, 0)
    dup_pairs = np.sum(sims > 0.95) / 2
    
    # 3. Format consistency
    formats = set()
    for ex in ds:
        # Detect chat template markers
        has_system = "<|system|>" in ex["instruction"] or "system" in ex.get("metadata", {})
        has_user = "<|user|>" in ex["instruction"] or "user" in ex.get("metadata", {})
        formats.add((has_system, has_user))
    
    return {
        "avg_instruction_tokens": np.mean(instr_lens),
        "avg_response_tokens": np.mean(resp_lens),
        "near_duplicate_pairs": int(dup_pairs),
        "format_variants": len(formats),
        "unique_instructions": len(set(ex["instruction"] for ex in ds)),
    }

Data quality vs. quantity: the tradeoff curve

Performance
    ^
    |                    ●●●●●●●●●●●●●●●●●●●● (high quality)
    |                 ●●●
    |              ●●●
    |           ●●●
    |        ●●●
    |     ●●●
    |  ●●●
    |●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● (low quality)
    +------------------------------------------------> Dataset size
         500    1K    2K    5K    10K   20K   50K  100K

Key inflection points:

• 500–1,000 examples: You’re in the “few-shot regime.” The model memorizes patterns. Works for narrow, deterministic tasks.
• 2,000–5,000 examples: Generalization emerges. The model learns the task structure, not just examples.
• 10,000–20,000 examples: Diminishing returns for SFT. Further gains come from preference optimization (DPO/ORPO) or better data, not more SFT data.
• 50,000+ examples: Only justified for continued pre-training, new languages, or massive domain shifts (e.g., training a legal model from a general base).

Task-specific guidelines

Classification and extraction (200–1,000 examples)

{"instruction": "Classify the support ticket urgency: low, medium, high, critical", "input": "User cannot login after password reset. Error: 'Invalid token'.", "output": "high"}
{"instruction": "Extract all medication names and dosages from the clinical note", "input": "Patient prescribed metformin 500mg BID and lisinopril 10mg daily.", "output": "[{\"medication\": \"metformin\", \"dosage\": \"500mg\", \"frequency\": \"BID\"}, {\"medication\": \"lisinopril\", \"dosage\": \"10mg\", \"frequency\": \"daily\"}]"}

Pitfall: Class imbalance. If 90% of your tickets are “low,” the model will predict “low” for everything. Fix: stratify sampling or use weighted loss.

Summarization and QA (1,500–5,000 examples)

Diversity matters more than volume. You need coverage across:

• Document lengths (short email → 50-page contract)
• Summary types (executive, bullet, technical, layperson)
• Domains (if multi-domain)
• Answer styles (extractive vs. abstractive)

# Ensure length diversity in summarization data
def check_length_coverage(dataset, target_buckets: list[tuple[int, int]]) -> dict:
    """
    target_buckets: [(min_tokens, max_tokens), ...] e.g. [(0, 512), (512, 2048), (2048, 8192)]
    """
    from collections import Counter
    bucket_counts = Counter()
    for ex in dataset:
        doc_len = len(ex["input"].split())
        for i, (lo, hi) in enumerate(target_buckets):
            if lo <= doc_len < hi:
                bucket_counts[i] += 1
                break
    return {f"bucket_{i} ({lo}-{hi})": bucket_counts[i] for i, (lo, hi) in enumerate(target_buckets)}

Code generation and reasoning (5,000–20,000 examples)

Code tasks are unforgiving. A single syntax error breaks the output. You need:

• Syntax-correct examples — Run a linter/formatter on every response
• Execution-verified examples — If possible, only keep examples where the code passes tests
• Library version awareness — Tag examples with dependency versions

# Minimal code quality filter
import ast
import subprocess
import tempfile

def validate_python_code(code: str, test_cases: list[str] = None) -> bool:
    # 1. Syntax check
    try:
        ast.parse(code)
    except SyntaxError:
        return False
    
    # 2. Optional: execution check
    if test_cases:
        with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
            f.write(code + "\n\n" + "\n".join(test_cases))
            fname = f.name
        try:
            result = subprocess.run(
                ["python", fname], capture_output=True, timeout=5
            )
            return result.returncode == 0
        except subprocess.TimeoutExpired:
            return False
    return True

Domain adaptation: legal, medical, finance (10,000–50,000+ examples)

Here you’re teaching vocabulary, reasoning patterns, and compliance constraints. The data must reflect the target distribution, not a generic sample. If you’re fine-tuning for contract review, your data should mirror the clause types, jurisdictions, and party structures you’ll see in production.

How to measure if you have enough data

Don’t guess. Run a learning curve with 3–4 data slices.

# learning_curve.py
import json
from pathlib import Path
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

def run_learning_curve(
    base_model: str,
    full_dataset_path: str,
    fractions: list[float] = [0.1, 0.25, 0.5, 1.0],
    eval_dataset_path: str = "eval.jsonl",
    output_dir: str = "learning_curve_runs",
):
    """
    Trains on increasing fractions of data, evaluates on held-out set.
    Returns the fraction where performance plateaus.
    """
    from datasets import load_dataset
    
    full_ds = load_dataset("json", data_files=full_dataset_path, split="train")
    eval_ds = load_dataset("json", data_files=eval_dataset_path, split="train")
    
    results = {}
    
    for frac in fractions:
        n = int(len(full_ds) * frac)
        train_ds = full_ds.shuffle(seed=42).select(range(n))
        
        model, tokenizer = FastLanguageModel.from_pretrained(
            model_name=base_model,
            max_seq_length=2048,
            load_in_4bit=True,
        )
        model = FastLanguageModel.get_peft_model(model, r=16, target_modules=["q_proj", "v_proj"])
        
        trainer = SFTTrainer(
            model=model,
            tokenizer=tokenizer,
            train_dataset=train_ds,
            eval_dataset=eval_ds,
            dataset_text_field="text",
            max_seq_length=2048,
            args=TrainingArguments(
                output_dir=f"{output_dir}/frac_{frac}",
                per_device_train_batch_size=2,
                gradient_accumulation_steps=4,
                max_steps=min(500, n // 8),  # ~1 epoch
                learning_rate=2e-4,
                fp16=False,
                bf16=True,
                logging_steps=10,
                evaluation_strategy="steps",
                eval_steps=50,
                save_strategy="no",
                report_to="none",
            ),
        )
        
        trainer.train()
        eval_metrics = trainer.evaluate()
        results[frac] = eval_metrics
        
        # Cleanup
        del model, trainer
        import torch; torch.cuda.empty_cache()
    
    with open(f"{output_dir}/learning_curve.json", "w") as f:
        json.dump(results, f, indent=2)
    
    return results

Interpretation: Plot eval_loss vs. fraction. If the curve flattens at 0.5, you don’t need the other 50%. If it’s still steep at 1.0, collect more data.

Evaluation metrics that matter

Common pitfalls

1. Contaminating eval with train data

# WRONG: random split on instruction level
train, eval = dataset.train_test_split(test_size=0.1)

# RIGHT: split on task/source/document level
# If multiple examples come from same document, keep them together
from datasets import DatasetDict

def split_by_source(dataset, source_key: str = "source_doc_id", test_size: float = 0.1):
    unique_sources = list(set(dataset[source_key]))
    test_sources = set(np.random.choice(unique_sources, int(len(unique_sources) * test_size), replace=False))
    
    train = dataset.filter(lambda ex: ex[source_key] not in test_sources)
    test = dataset.filter(lambda ex: ex[source_key] in test_sources)
    return DatasetDict({"train": train, "test": test})

2. Ignoring token length distribution

If 90% of your training examples are < 512 tokens but production inputs are 4K tokens, the model never learns to attend over long contexts. Match the length distribution or use length-stratified sampling.

# Length-stratified sampler for DataLoader
from torch.utils.data import Sampler

class LengthStratifiedSampler(Sampler):
    def __init__(self, dataset, batch_size: int, num_buckets: int = 10):
        lengths = [len(ex["input_ids"]) for ex in dataset]
        self.buckets = [[] for _ in range(num_buckets)]
        max_len = max(lengths)
        for idx, l in enumerate(lengths):
            bucket_idx = min(int(l / max_len * num_buckets), num_buckets - 1)
            self.buckets[bucket_idx].append(idx)
        self.batch_size = batch_size
    
    def __iter__(self):
        batches = []
        for bucket in self.buckets:
            np.random.shuffle(bucket)
            for i in range(0, len(bucket), self.batch_size):
                batch = bucket[i:i + self.batch_size]
                if len(batch) == self.batch_size:
                    batches.append(batch)
        np.random.shuffle(batches)
        for batch in batches:
            yield batch
    
    def __len__(self):
        return sum(len(b) // self.batch_size for b in self.buckets)

3. Training on bad responses

Including low-quality responses (hallucinations, refusal when it shouldn’t, wrong format) teaches the model to reproduce them. Filter aggressively:

# Automated response quality filters
def filter_responses(dataset, min_score: float = 0.7) -> Dataset:
    """
    Uses a small reward model or LLM-as-judge to score responses.
    Keeps only examples above threshold.
    """
    # Placeholder: in practice, use a trained reward model or
    # a strong LLM (e.g., GPT-4o-mini) to score each (instruction, response)
    scored = []
    for ex in dataset:
        score = score_response_quality(ex["instruction"], ex["output"])
        if score >= min_score:
            scored.append(ex)
    return Dataset.from_list(scored)

def score_response_quality(instruction: str, response: str) -> float:
    # Implement with your preferred judge
    # Return 0.0-1.0
    pass

4. Overfitting to format, not task

If every training example uses the exact same prompt template, the model learns the template, not the task. Vary the phrasing:

# Template variation for instruction diversity
TEMPLATES = [
    "{instruction}\n\nInput: {input}\nOutput:",
    "Task: {instruction}\n\n{input}\n\nAnswer:",
    "Below is an instruction. Complete the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:",
    "You are an expert. {instruction}\n\nContext: {input}\n\nResponse:",
]

def apply_random_template(ex):
    template = np.random.choice(TEMPLATES)
    ex["text"] = template.format(instruction=ex["instruction"], input=ex.get("input", ""))
    ex["text"] += ex["output"] + tokenizer.eos_token
    return ex

Practical workflow: from zero to trained model

Week 1: Define and collect

1. Write 50–100 gold examples by hand — This forces you to clarify the task, edge cases, and output format.
2. Run few-shot evaluation on your base model with those 50 examples. Establish a baseline.
3. If few-shot hits your quality bar, stop. You don’t need fine-tuning.

Week 2: Scale and clean

1. Generate synthetic data using a stronger model (GPT-4o, Claude 3.5 Sonnet) prompted with your gold examples.
2. Filter synthetics with the quality checks above (syntax, execution, judge score).
3. Deduplicate against your gold set and within synthetics (embedding cosine > 0.95).
4. Target 2,000–5,000 examples for first training run.

Week 3: Train and evaluate

1. Run learning curve at 25%, 50%, 75%, 100%.
2. Evaluate on held-out gold set (never seen during training or synthetic generation).
3. Error analysis: Categorize failures — format errors, factual errors, reasoning gaps, refusal.

Week 4: Iterate or ship

• If learning curve plateaus early: You have enough data. Ship the smallest checkpoint that meets quality.
• If curve still climbing: Generate more data targeting error categories.
• If quality gap persists: Consider DPO/ORPO with preference pairs, not more SFT data.

When to stop collecting

You have enough data when any of these hold:

1. Learning curve plateaus (eval loss change < 0.01 over 2x data increase)
2. Eval metrics match or exceed your few-shot baseline with a stronger model
3. Error analysis shows failures are inherent ambiguity (human annotators disagree), not model capability
4. Cost of next 1,000 examples > expected value of marginal improvement

Bottom line: For most teams, the answer to how much data to fine-tune is 2,000–5,000 verified examples for instruction tuning, 5,000–20,000 for code or reasoning, and 10,000+ for deep domain adaptation. But the only number that matters is the one your learning curve gives you. Run the experiment.