Common mistakes when fine-tuning an LLM

Fine-tuning an LLM looks deceptively simple: grab a dataset, pick a base model, run a training loop, and ship. In practice, the gap between a working prototype and a reliable production model is littered with fine-tuning mistakes that only surface under load, on edge cases, or after a silent distribution shift. Most teams don’t fail because they chose the wrong optimizer; they fail because they treated fine-tuning as a one-off script instead of a disciplined engineering process.

1. Skipping a rigorous baseline before training

You cannot measure improvement without a fixed, reproducible baseline. Too many teams fine-tune against a moving target — different prompt templates, different sampling parameters, or even different base model checkpoints across experiments. The result is a model that “feels better” on vibes but regresses on the metrics that matter.

Establish a frozen evaluation harness first. Pin the base model version, the prompt format, temperature, top-p, and max tokens. Run a held-out test set of 200–500 examples covering your core tasks and failure modes. Log exact scores (exact match, F1, BLEU, or your task-specific metric) before you touch a single training hyperparameter. Only then can you attribute delta to your fine-tuning run.

# evaluation/harness.py
from dataclasses import dataclass
from typing import Callable
import json

@dataclass(frozen=True)
class EvalConfig:
    model_id: str           # e.g. "meta-llama/Meta-Llama-3-8B-Instruct"
    prompt_template: str    # frozen jinja2 template
    temperature: float = 0.0
    top_p: float = 1.0
    max_tokens: int = 512

def run_baseline(config: EvalConfig, dataset: list[dict], metric_fn: Callable) -> dict:
    # deterministic generation, fixed seed
    results = []
    for ex in dataset:
        prompt = config.prompt_template.format(**ex)
        completion = generate(prompt, config)  # your inference wrapper
        results.append({"prompt": prompt, "completion": completion, **ex})
    return {"config": config.__dict__, "scores": metric_fn(results)}

2. Treating data quality as someone else’s problem

The single highest-leverage lever in fine-tuning is data quality, yet it’s the step most often delegated or automated away. Noisy labels, duplicate examples, PII leakage, and format inconsistencies all propagate directly into the model. A 10k-example dataset with 15% label noise will underperform a clean 2k-example set every time.

Invest in a data pipeline you can audit. Deduplicate at the string and semantic level. Enforce a schema with required fields and validation rules. Sample 5–10% of every batch for manual review. Track provenance: source, annotator, version, and any transformations applied. If you cannot reproduce the exact training set from a commit hash, you don’t have a dataset — you have a liability.

# data/validate.py
import hashlib
from pydantic import BaseModel, field_validator

class TrainExample(BaseModel):
    id: str
    messages: list[dict]  # [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
    source: str
    annotator_id: str
    version: int

    @field_validator("messages")
    @classmethod
    def validate_format(cls, v):
        if not v or v[-1]["role"] != "assistant":
            raise ValueError("Last message must be assistant")
        return v

def dedupe(examples: list[TrainExample]) -> list[TrainExample]:
    seen = set()
    unique = []
    for ex in examples:
        key = hashlib.sha256(ex.messages[-1]["content"].encode()).hexdigest()
        if key not in seen:
            seen.add(key)
            unique.append(ex)
    return unique

3. Ignoring tokenization mismatches between base and fine-tuned models

If your fine-tuning run uses a different tokenizer than the base model — or if you extend the vocabulary without re-embedding — you introduce silent degradation. The model sees token IDs it never encountered during pre-training, or the embedding matrix grows with randomly initialized rows that destabilize early training steps.

Verify tokenizer identity before you start. tokenizer.vocab_size, tokenizer.get_vocab(), and special token mappings must match exactly. If you must add tokens (domain-specific terms, control codes), use resize_token_embeddings and initialize new embeddings as the mean of existing embeddings, not random normal. Run a forward pass on a handful of prompts and compare logits for the first 10 tokens against the base model; they should be nearly identical before training begins.

# Quick sanity check
python -c "
from transformers import AutoTokenizer
base = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B-Instruct')
ft = AutoTokenizer.from_pretrained('./my-fine-tuned-tokenizer')
assert base.vocab_size == ft.vocab_size
assert base.get_vocab() == ft.get_vocab()
print('Tokenizers match')
"

4. Using LoRA defaults without tuning rank and alpha

LoRA is not a free lunch. The default r=8, alpha=16 works for some tasks but fails catastrophically on others — especially when the target domain diverges significantly from pre-training data. Too low a rank bottlenecks capacity; too high overfits and negates the parameter-efficiency advantage. Alpha scales the effective learning rate of the adapter, and the ratio alpha/r matters more than either alone.

Treat r and alpha as hyperparameters. Sweep r ∈ {8, 16, 32, 64, 128} with alpha = 2*r as a starting heuristic, then vary alpha/r ∈ {0.5, 1, 2, 4}. Target modules matter too: q_proj, v_proj is the minimal set; adding k_proj, o_proj, gate_proj, up_proj, down_proj increases capacity but also memory. Log trainable parameter count and GPU memory per configuration. Pick the smallest config that matches full fine-tuning performance on your validation set.

# lora_sweep.py
from peft import LoraConfig, TaskType

def lora_configs():
    for r in [8, 16, 32, 64, 128]:
        for alpha_ratio in [0.5, 1, 2, 4]:
            alpha = int(r * alpha_ratio)
            yield LoraConfig(
                task_type=TaskType.CAUSAL_LM,
                r=r,
                lora_alpha=alpha,
                lora_dropout=0.05,
                target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                "gate_proj", "up_proj", "down_proj"],
                bias="none",
            )

5. Training until training loss plateaus instead of validating

Training loss is a proxy, not the objective. It will keep decreasing long after validation metrics peak, especially with large models on small datasets. Early stopping on validation loss (or your task metric) is non-negotiable. Without it, you ship an overfitted model that hallucinates confidently on out-of-distribution inputs.

Reserve a true hold-out set — not a split from your training data, but a separately curated slice that reflects production distribution. Evaluate every N steps (N = max(1, len(train_dataset) // (batch_size * 10))). Stop when the metric hasn’t improved for 3–5 evaluations. Save the best checkpoint, not the last. If you’re using gradient accumulation, make sure evaluation frequency accounts for effective batch size.

# trainer config snippet (HF Trainer / TRL)
training_arguments:
  evaluation_strategy: "steps"
  eval_steps: 50
  save_strategy: "steps"
  save_steps: 50
  load_best_model_at_end: true
  metric_for_best_model: "eval_f1"
  greater_is_better: true
  early_stopping_patience: 3

6. Neglecting chat template and special token alignment

Instruction-tuned base models expect a specific chat template (system, user, assistant roles with particular delimiters). If your fine-tuning data uses a different format — or worse, no format at all — the model learns to ignore the template it was aligned to. At inference time, you’ll feed the correct template but the model will have unlearned how to use it.

# Verify template alignment
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is 2+2?"},
    {"role": "assistant", "content": "4"},
]
rendered = tokenizer.apply_chat_template(messages, tokenize=False)
print(repr(rendered))
# '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n4<|eot_id|>'

7. Skipping quantization-aware fine-tuning for deployment targets

If you plan to deploy at 4-bit or 8-bit quantization (QLoRA, AWQ, GPTQ), fine-tuning in full precision then quantizing post-hoc often degrades quality more than necessary. The quantization error compounds with the fine-tuning adaptation, especially on sensitive layers like the output projection. Quantization-aware training (QAT) or at least quantization-aware LoRA (QALoRA) closes this gap.

If you’re using QLoRA, you’re already training with 4-bit base weights — but the adapters train in fp16/bf16. For production quantization (AWQ/GPTQ), run a short QAT phase: freeze the base, quantize it with your target calibration data, then fine-tune only the adapters (or a small subset of layers) with straight-through estimator gradients. Even 500–1000 steps recovers most of the post-quantization quality drop.

# Example: QLoRA training (already quantization-aware for base)
# But if deploying with AWQ, add a QAT phase:
python -m llmcompressor.finetune \
  --model ./my-lora-merged \
  --dataset calib_data \
  --recipe "quantization: awq, bits: 4, group_size: 128" \
  --output_dir ./my-awq-model \
  --num_train_epochs 0.1 \
  --learning_rate 1e-5

8. Shipping without an eval regression gate in CI

A model that passes manual inspection today will regress tomorrow when a dependency updates, a dataset shifts, or a teammate merges a “harmless” preprocessing change. Without an automated gate, you discover regressions in production — or worse, you don’t discover them at all.

Add a CI step that runs your frozen evaluation harness against the candidate model artifact. Fail the build if any core metric drops more than a defined threshold (e.g., 1% absolute F1, 0.5 BLEU). Store evaluation artifacts (predictions, scores, confusion matrices) as build artifacts for auditability. Tag the model artifact with the git commit, dataset version, and training config hash. This turns fine-tuning from an artisanal process into a reproducible pipeline.

# .github/workflows/eval-gate.yml
name: Model Evaluation Gate
on:
  workflow_dispatch:
    inputs:
      model_artifact:
        required: true
      dataset_version:
        required: true

jobs:
  evaluate:
    runs-on: gpu-runner
    steps:
      - uses: actions/checkout@v4
      - name: Download model
        run: ./scripts/pull_model.sh ${{ inputs.model_artifact }}
      - name: Run evaluation
        run: |
          python -m evaluation.harness \
            --model ./model \
            --dataset-version ${{ inputs.dataset_version }} \
            --output eval_results.json
      - name: Check thresholds
        run: |
          python -c "
          import json, sys
          with open('eval_results.json') as f:
              r = json.load(f)
          # Example thresholds
          assert r['f1'] >= 0.82, f'F1 {r[\"f1\"]} below threshold 0.82'
          assert r['exact_match'] >= 0.75, f'EM {r[\"exact_match\"]} below threshold 0.75'
          print('All thresholds passed')
          "
      - name: Upload eval artifacts
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: eval_results.json

Summary: fine-tuning mistakes checklist

#	Mistake	Signal you have it	Fix
1	No frozen baseline	“It feels better” but no numbers	Pin model, prompt, params; run held-out eval first
2	Data quality delegation	Silent failures on edge cases	Schema validation, dedupe, provenance tracking, manual audit
3	Tokenizer drift	Weird tokens, degraded first-token logits	Verify vocab equality; mean-init new embeddings
4	LoRA defaults	Overfit or underfit vs full FT	Sweep `r`, `alpha`, target modules; pick smallest working config
5	Training loss stopping	Val metric peaks then drops	Early stop on hold-out val; save best checkpoint
6	Chat template mismatch	Model ignores system prompts at inference	Mirror base template exactly in training data
7	Post-hoc quantization	Quality drop after AWQ/GPTQ	QAT/QALoRA phase for deployment quantization
8	No CI gate	Regressions found in prod	Automated eval in CI with hard thresholds

Fine-tuning mistakes compound silently. A tokenizer mismatch here, a skipped baseline there, and you end up with a model that passes smoke tests but fails on the long tail your users actually hit. The discipline above is boring — it’s schema validation, config pinning, CI gates — but it’s the difference between a demo and a dependency your product can rely on.