Full fine-tuning vs parameter-efficient fine-tuning

The choice between full fine-tuning vs peft shapes your entire training pipeline: GPU budget, iteration speed, model quality ceiling, and operational complexity. Full fine-tuning updates every weight; PEFT methods like LoRA, QLoRA, and adapters freeze the base model and train a tiny fraction of parameters. That distinction cascades into every downstream decision.

What each approach actually does

Full fine-tuning backpropagates through the entire model. Every transformer block, attention head, and MLP layer receives gradient updates. The optimizer maintains state — typically AdamW moments — for every parameter. For a 7B parameter model, that means 7B weights plus 14B optimizer states (fp32) plus gradients, all resident in GPU memory simultaneously.

PEFT freezes the base model and injects trainable modules. LoRA adds low-rank decomposition matrices (A and B) to attention projections. A rank-64 LoRA on a 7B model trains roughly 0.5% of parameters. QLoRA quantizes the base model to 4-bit NF4, then applies LoRA — fitting a 7B model on a single 24GB GPU. Adapters insert bottleneck layers between transformer blocks. Prefix tuning prepends trainable vectors to the key-value cache. Prompt tuning only optimizes soft prompt embeddings.

The mathematical difference is rank. Full fine-tuning operates in the full parameter space. LoRA constrains updates to a low-rank subspace: ΔW = BA where B ∈ ℝ^(d×r), A ∈ ℝ^(r×d), r ≪ d. This subspace hypothesis — that adaptation lives in a low-dimensional manifold — holds empirically for many tasks but fails when the target distribution diverges sharply from pretraining.

Compute and memory profile

Full fine-tuning a 7B model demands multi-GPU with ZeRO-3 or FSDP. You shard optimizer states, gradients, and parameters across GPUs. Communication overhead scales with cluster size. PEFT fits on fewer devices — often one — eliminating distributed coordination complexity. QLoRA’s 4-bit quantization adds dequantization overhead during forward/backward passes (~15-20% slower per step), but the memory savings let you run larger batch sizes, often netting higher throughput.

Storage differs too. A full fine-tuned 7B checkpoint is ~14 GB (bf16). A LoRA adapter is ~70 MB. You can version dozens of LoRA adapters for the price of one full checkpoint. Merging LoRA into base weights (W + BA) produces a standalone model with zero inference overhead — but merging is destructive; you lose the ability to swap or compose adapters.

Capability ceiling and failure modes

Full fine-tuning can rewrite the model’s fundamental behavior. It learns new languages, acquires new reasoning patterns, absorbs massive domain corpora (legal, biomedical, code), and adapts to distribution shifts that PEFT cannot bridge. Continued pretraining on 1T tokens is full fine-tuning by another name.

PEFT excels at style transfer, instruction following, classification heads, and domain adaptation where the target distribution shares structure with pretraining. It struggles when:

• The task requires new factual knowledge not in pretraining (LoRA memorizes poorly)
• The output space is structurally different (e.g., pretraining on English, fine-tuning on a new script)
• You need compositional generalization beyond the training distribution
• The base model has strong priors that conflict with the target task (safety alignment fighting domain specificity)

Empirically, LoRA matches full fine-tuning on Super-NaturalInstructions, AlpacaEval, and many classification benchmarks at 7B-13B scale. The gap widens at 70B+ on knowledge-intensive tasks (MMLU, TriviaQA) and when training from scratch on domain corpora. QLoRA adds ~1-2% degradation vs bf16 LoRA from quantization noise.

Training dynamics and hyperparameter sensitivity

Full fine-tuning is surprisingly robust. Learning rates of 1e-5 to 2e-5 with cosine decay work across domains. Weight decay 0.01-0.1. Batch sizes 128-512 tokens per GPU. The main failure mode is catastrophic forgetting — the model loses pretrained capabilities. Mitigations: lower LR, shorter training, KL regularization against a frozen reference, or replay buffers.

PEFT is hyperparameter-sensitive in different ways. LoRA rank (r) and alpha (scaling) interact: effective LR = base_lr × (alpha / r). Common defaults: r=64, alpha=16 (effective LR 4× base). Target modules matter — q_proj, v_proj only is standard; adding k_proj, o_proj, gate_proj, up_proj, down_proj helps on difficult tasks but increases params. Dropout on LoRA layers (0.05-0.1) prevents overfitting on small datasets.

QLoRA introduces quantization hyperparameters: nf4 vs fp4, double quantization, compute dtype (bf16 vs fp16). NF4 with double quantization and bf16 compute is the stable default. Gradient accumulation steps must be tuned — small per-device batches with large accumulation simulate larger batches but change optimizer step frequency.

# LoRA config that works for most 7B-13B instruction tuning
from peft import LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=64,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
)

Inference latency and serving

Merged LoRA (W ← W + BA) has identical inference latency to the base model. No overhead. Unmerged LoRA requires two additional matmuls per adapted layer: x @ A.T @ B.T. At r=64, d=4096, that’s ~2× FLOPs for the adapted projections — negligible at batch size 1, measurable at high throughput.

Full fine-tuned models serve identically to base models. No architectural changes.

PEFT enables adapter composition at inference time: you can load multiple LoRA adapters and route requests to different adapters without reloading the base model. This is a genuine operational advantage for multi-tenant serving. Some inference engines (vLLM, TGI, TensorRT-LLM) support dynamic LoRA loading with batched requests targeting different adapters.

Tooling and ecosystem maturity

Full fine-tuning tooling centers on distributed training frameworks: FSDP (PyTorch native), DeepSpeed (Microsoft), Megatron-LM (NVIDIA), YaFSDP. Checkpointing, resumption, and sharding are solved but complex. Hyperparameter sweeps require cluster access.

PEFT tooling is concentrated in Hugging Face PEFT, with integrations in Axolotl, LLaMA-Factory, Unsloth, and TRL. Unsloth deserves specific mention — it fuses kernels for 2-5× faster LoRA/QLoRA training with lower memory. Axolotl provides YAML-driven configs that abstract away boilerplate. The ecosystem moves fast; best practices from six months ago are often suboptimal.

Debugging differs. Full fine-tuning: you monitor loss curves, gradient norms, weight norms across all layers. PEFT: you monitor the same plus LoRA-specific metrics — singular value decay of BA, effective rank utilization, adapter norm growth. LoRA weights that collapse to near-zero indicate underfitting; exploding norms indicate instability.

When the limits bite

Full fine-tuning hits hard limits at:

• Model scale: 70B+ requires 16+ H100s with expert parallelism knowledge
• Data scale: Multi-epoch training on small datasets overfits; you need data diversity
• Iteration speed: 1-4 hours per epoch on 7B at 8×A100; experimentation is slow
• Cost: $500-2000 per 7B run on cloud spot; 70B is $10k+

PEFT hits limits at:

• Distribution shift: Cannot learn new languages, scripts, or reasoning paradigms
• Knowledge injection: Poor at memorizing new facts (use RAG instead)
• Compositional tasks: Struggles when output structure diverges from pretraining
• Rank saturation: Increasing r beyond 128-256 yields diminishing returns; you’re back to full fine-tuning compute

A practical boundary: if your validation loss plateaus while training loss drops, and increasing r doesn’t help, the task likely needs full fine-tuning. Conversely, if full fine-tuning overfits within 0.5 epochs on your dataset, PEFT’s regularization-by-constraint is the better inductive bias.

Which to choose

Choose full fine-tuning when:

• Continued pretraining on domain corpora (>100B tokens)
• New language or script acquisition
• Fundamental capability gaps (reasoning, coding, tool use) that PEFT cannot bridge
• You have 8+ GPUs and 2+ weeks of engineering time for distributed training
• The model will be a long-lived foundation for multiple downstream adapters

Choose LoRA (bf16) when:

• Instruction tuning, chat, style transfer, classification
• Domain adaptation where pretraining covers the vocabulary and concepts
• 1-4 GPUs available (A100 40/80GB, H100)
• You need to maintain multiple task-specific variants
• Iteration speed matters — experiments in hours, not days

Choose QLoRA when:

• Single consumer GPU (24-48GB VRAM) or limited cloud budget
• Prototyping on 7B-70B before committing to full fine-tuning
• Serving multiple adapters on shared base model infrastructure
• Acceptable to trade 1-2% quality for 4-8× memory reduction

Choose adapters/prefix tuning when:

• You need zero-latency switching at inference (prefix tuning shares KV cache)
• Parameter budget is extremely tight (<0.1%)
• Composing many small adaptations (multilingual, multi-domain)

Default heuristic: Start with QLoRA r=64 on your largest feasible model. If validation metrics saturate below target and you’ve exhausted data quality, hyperparameters, and rank — then provision for full fine-tuning. Most teams never need to cross that threshold.