LoRA vs full fine-tuning: costs and tradeoffs

When you need to adapt a foundation model to your domain, the first architectural decision is LoRA vs full fine-tuning. The choice cascades into GPU budget, iteration speed, serving infrastructure, and how much model behavior you can actually change. This post breaks down the tradeoffs across six dimensions so you can pick the right tool without running a month-long bake-off.

What each method actually does

Full fine-tuning updates every weight in the model. You backpropagate through the entire parameter count — 7B, 70B, 405B — and produce a new checkpoint of identical size. The optimizer state, gradients, and activations for all layers must fit in VRAM during training.

LoRA (Low-Rank Adaptation) freezes the base model and injects trainable low-rank matrices into targeted modules, typically the attention projection layers. For a 7B model with rank 64 on q/k/v/o projections, you train roughly 0.5% of parameters. The base weights never move; you only materialize the adapter weights at inference by adding them to the frozen weights: W' = W + BA.

QLoRA adds 4-bit quantization of the base model via NF4, paging optimizer states to CPU, and double quantization of constants. This pushes the VRAM floor low enough to fine-tune 70B models on a single 48 GB GPU.

# LoRA config example (peft)
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 7,241,331,712 || trainable%: 0.0579%

Capabilities: what behavior can you change?

Full fine-tuning can rewrite any capability the model possesses: new languages, reasoning styles, tool-use formats, or fundamental knowledge updates. It can also unlearn behaviors — safety alignment, refusal patterns, or hallucination tendencies — because every weight is plastic.

LoRA is additive. It excels at style transfer, format adherence, domain vocabulary, and task-specific reasoning patterns that compose on top of existing capabilities. It struggles when the target behavior contradicts strong priors in the frozen weights. You cannot easily make a model “forget” training data or unlearn a refusal direction with LoRA alone; the base weights still project the original features.

Empirically, LoRA matches full fine-tuning on classification, extraction, and structured generation tasks. It lags on knowledge-intensive tasks requiring new factual recall and on complex reasoning where the optimal solution requires coordinated changes across many layers.

Price and cost model

The VRAM difference is the single biggest practical constraint. Full fine-tuning a 70B model requires a 16×H100 node minimum (80 GB each) with tensor/pipeline parallelism. LoRA on the same model fits on one H100 or even an A100 80 GB. For most teams, this alone decides the question.

Cloud pricing makes this concrete: an 8×H100 node runs ~$30–40/hr. Full fine-tuning 70B for one epoch on 1T tokens is a five-figure experiment. LoRA on 70B is a few hundred dollars.

Latency and throughput

At serving time, merged LoRA adds zero latency — you materialize W + BA once and serve the merged weights. Unmerged multi-adapter serving (swapping adapters per request) adds a small kernel launch overhead per layer but avoids loading multiple full models.

Full fine-tuned models serve identically to base models: same latency, same throughput, same KV cache footprint. No adapter bookkeeping.

If you serve dozens of fine-tuned variants, LoRA wins on GPU memory: one base model + many tiny adapters versus N full checkpoints. This matters for multi-tenant SaaS or per-customer customization.

# Merge LoRA into base for zero-overhead serving
python -m peft.merge_adapter \
  --base_model meta-llama/Llama-3-8B \
  --adapter_path ./my-lora-adapter \
  --output_path ./merged-model

Ergonomics and iteration speed

LoRA enables rapid iteration. You can train an adapter in 30 minutes on a single GPU, evaluate, tweak hyperparameters, and retrain. The feedback loop is tight enough for daily experimentation.

Full fine-tuning demands distributed training infrastructure: DeepSpeed ZeRO-3, FSDP, or Megatron-LM. Debugging gradient overflow, communication hangs, or optimizer state sharding across nodes consumes engineering time. Checkpointing and resuming a 70B run is nontrivial.

Hyperparameter sensitivity differs too. LoRA has three main knobs: rank (r), alpha, and dropout. Full fine-tuning exposes the entire optimizer landscape — learning rate schedules, weight decay, gradient clipping, warmup steps, batch size scaling laws. More degrees of freedom means more tuning surface.

# Typical LoRA sweep (yaml for wandb/optuna)
parameters:
  r:
    values: [8, 16, 32, 64, 128]
  lora_alpha:
    values: [16, 32, 64, 128, 256]
  lora_dropout:
    values: [0.0, 0.05, 0.1]
  learning_rate:
    values: [1e-4, 2e-4, 5e-4, 1e-3]

Ecosystem and tooling

Full fine-tuning tooling centers on distributed training frameworks: Hugging Face trl, llama-factory, axolotl, NVIDIA NeMo, MosaicML Composer. These handle sharding, activation checkpointing, and mixed precision. The ecosystem assumes cluster access.

LoRA tooling is lighter and more accessible: peft + trl SFTTrainer, unsloth for optimized kernels, llama-factory UI, axolotl configs. unsloth in particular delivers 2–3× throughput on consumer GPUs via fused kernels and is a force multiplier for single-GPU workflows.

Model hub support favors LoRA. Hugging Face Hub natively renders adapter cards, supports peft loading with from_pretrained(model_id, adapter_name="..."), and enables multi-adapter inference via set_adapter(). Full fine-tuned models are just large repos — no special affordances.

Limits and failure modes

LoRA fails when:

• The task requires knowledge not in the base model (new programming language, recent events, proprietary corpus)
• The target behavior contradicts strong alignment (refusal directions, safety filters)
• You need to change tokenization or vocabulary
• Rank is too low for the task complexity (symptom: training loss plateaus above base model eval loss)

Full fine-tuning fails when:

• You lack cluster compute (the hard constraint)
• Catastrophic forgetting destroys base capabilities you still need
• Training instability wastes the budget (loss spikes, gradient explosion, NaNs)
• You need to serve many variants — each full model consumes a full GPU deployment

Catastrophic forgetting is real. Full fine-tuning on a narrow domain can degrade general reasoning, instruction following, and multilingual ability. LoRA preserves the base model by construction, though high-rank LoRA on aggressive learning rates can still drift.

Which to choose

Start with LoRA (QLoRA) if:

• You have ≤ 48 GB VRAM per GPU (single A100/H100 or consumer 3090/4090)
• The task is style, format, extraction, classification, or reasoning on existing knowledge
• You need to serve multiple variants or iterate daily
• Budget is measured in hundreds, not thousands, of dollars
• You want to experiment with ranks 16–128 before committing

Move to full fine-tuning if:

• You have a 16+ GPU cluster and distributed training expertise
• The task requires new factual knowledge or language acquisition
• LoRA at rank 256 with tuned hyperparameters still underperforms base model on evals
• You need to unlearn or fundamentally redirect model behavior
• The serving footprint of one model per variant is acceptable

Hybrid approach (common in practice):

1. Run LoRA sweeps first — cheap, fast, establishes ceiling
2. If LoRA saturates below target, allocate cluster for full fine-tuning
3. Use LoRA adapters on top of a periodically full-fine-tuned base (continual learning pattern)

# Continual pattern: full fine-tune base quarterly, LoRA daily
# Quarterly: full FT on accumulated data -> new base checkpoint
# Daily: LoRA on new data atop current base -> adapter
# Serve: base + active adapter(s)

One more operational note

If you route inference through a gateway that supports per-request adapter selection, you can serve a single base model with dozens of LoRA adapters hot-swapped per tenant or task. This avoids the “one GPU per fine-tuned model” tax entirely. The gateway merges adapters on the fly or keeps multiple adapters resident in VRAM, adding microseconds per request. For multi-tenant LLM products, this architecture pays for itself quickly.