LoRA (Low-Rank Adaptation) freezes a pretrained model’s weights and injects trainable low-rank matrices into selected layers, reducing trainable parameters by orders of magnitude while preserving most full fine-tuning performance. Instead of updating all 7B or 70B parameters, you train two small matrices per adapted layer — typically adding 0.1–1% extra parameters. The result: fine-tuning that fits on a single GPU, finishes in hours instead of days, and produces adapters you can swap at inference time without reloading the base model.
How LoRA works
The core insight: weight updates during fine-tuning tend to have low intrinsic rank. A full weight matrix $W \in \mathbb{R}^{d \times k}$ gets an additive update $\Delta W$. LoRA constrains $\Delta W = BA$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with $r \ll \min(d, k)$. During training, only $A$ and $B$ receive gradients; $W$ stays frozen.
# Conceptual LoRA layer (PyTorch-style)
class LoRALinear(nn.Module):
def __init__(self, in_features, out_features, rank=16, alpha=32, dropout=0.0):
super().__init__()
self.base = nn.Linear(in_features, out_features, bias=False)
self.base.weight.requires_grad = False # frozen
self.lora_A = nn.Parameter(torch.zeros(rank, in_features))
self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
self.scaling = alpha / rank
self.dropout = nn.Dropout(dropout)
# init: A ~ N(0, 1), B = 0 so initial output = base only
nn.init.normal_(self.lora_A, std=1/rank)
nn.init.zeros_(self.lora_B)
def forward(self, x):
base_out = self.base(x)
lora_out = (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scaling
return base_out + lora_outAt inference, you can fuse the adapter into the base weights: $W’ = W + \frac{\alpha}{r}BA$. This adds zero latency — the model runs at base speed. You can also keep adapters separate and hot-swap them per request, which is useful for multi-tenant serving.
Where the matrices go
Most implementations target attention projections (q, k, v, o) and sometimes MLP up/down projections. The original LoRA paper adapted only query and value projections in attention; later work (LoRA+, DoRA, VeRA) showed adapting all linear layers often helps. A typical LLaMA-7B with rank 16 on all linear layers adds ~4.2M parameters (0.06% of base).
{
"model": "llama-7b",
"base_params": 6.74e9,
"lora_config": {
"rank": 16,
"alpha": 32,
"target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
},
"trainable_params": 4_194_304,
"trainable_pct": 0.062
}Why it matters
Memory and compute
Full fine-tuning a 7B model in FP16 needs ~56 GB GPU memory for optimizer states alone (AdamW: 2x params for moments + gradients + model). LoRA at rank 16 drops that to ~6–8 GB — feasible on a single A10G or 24 GB consumer card. Training time scales similarly: fewer parameters, less gradient computation, smaller optimizer step.
Adapter composability
Because adapters are additive and small (tens of MB), you can:
- Store dozens of task-specific adapters alongside one base model
- Load the base once, swap adapters per request
- Merge multiple adapters via weighted sum: $W’ = W + \sum_i \lambda_i \Delta W_i$
This enables practical multi-tenant serving without per-tenant model copies.
Portability
A LoRA adapter trained on one base model checkpoint transfers to other checkpoints of the same architecture (e.g., LLaMA-2-7B → LLaMA-2-7B-Chat) with minimal degradation. You can also stack LoRA on top of other PEFT methods (prefix tuning, adapters) — though stacking adds complexity.
Rank selection and scaling
Rank $r$ controls capacity. Common values: 8, 16, 32, 64, 128. Higher rank = more capacity but diminishing returns and more memory. The scaling factor $\alpha/r$ matters: $\alpha$ is typically set to $2r$ or $r$, making effective learning rate for LoRA parameters $\eta \cdot \alpha/r$. If you change rank, adjust $\alpha$ proportionally to keep effective LR stable.
# Practical rank sweep (pseudocode)
for rank in [8, 16, 32, 64, 128]:
config = LoraConfig(
r=rank,
lora_alpha=rank * 2, # alpha = 2r is a solid default
target_modules=["q_proj", "v_proj"], # start minimal
lora_dropout=0.05,
bias="none",
)
trainer = Trainer(model, config, ...)
trainer.train()
eval_metrics = trainer.evaluate()
log(rank, eval_metrics)Empirical pattern: rank 16–32 often hits 95%+ of full fine-tuning performance on downstream tasks. Rank 8 works for simple style/format adaptation. Rank 64+ rarely justified unless the task is far from pretraining distribution (e.g., coding on a base model never exposed to code).
LoRA+ and stabilized variants
LoRA+ uses different learning rates for $A$ and $B$ (typically $\eta_B = \eta_A / \text{rank}$) to balance gradient magnitudes. DoRA (Weight-Decomposed Low-Rank Adaptation) separates magnitude and direction updates, matching full fine-tuning more closely at similar parameter count. VeRA shares frozen random projections across layers and only learns per-layer scaling vectors — extremely parameter-efficient (0.01%) but slightly lower ceiling.
Concrete example: instruction tuning LLaMA-3-8B
Scenario: adapt a base LLaMA-3-8B to follow chat formatting and refuse unsafe requests. Dataset: 50k examples (Alpaca + safety mix). Hardware: 1× H100 80GB.
# Using Hugging Face PEFT + TRL
python train.py \
--model_name_or_path meta-llama/Meta-Llama-3-8B \
--dataset_name tatsu-lab/alpaca \
--peft_config r=32,lora_alpha=64,target_modules=q_proj,v_proj,k_proj,o_proj \
--learning_rate 2e-4 \
--num_train_epochs 3 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--bf16 \
--output_dir llama3-8b-lora-alpacaResults typical for this setup:
- Trainable params: ~6.3M (0.08%)
- Peak GPU memory: ~18 GB (vs ~80 GB for full FT)
- Training time: ~2.5 hours
- Eval: MT-Bench 7.2 (base: 6.1), GSM8K 48% (base: 32%)
The adapter checkpoint is ~25 MB. At inference, you load the base model once and apply the adapter per request — or merge it for zero-overhead deployment.
Merging for deployment
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", torch_dtype=torch.bfloat16)
adapter = PeftModel.from_pretrained(base, "llama3-8b-lora-alpaca")
merged = adapter.merge_and_unload() # fuses LoRA into base weights
merged.save_pretrained("llama3-8b-alpaca-merged")The merged model runs at native speed with no PEFT dependency. Trade-off: you lose hot-swapping. For single-tenant production, merge. For multi-tenant or A/B testing, keep separate.
Common misconceptions
“LoRA always matches full fine-tuning”
False. On tasks requiring substantial new knowledge (e.g., learning a new language, memorizing large proprietary corpora), LoRA underperforms full fine-tuning. The low-rank constraint limits how much the model’s internal representations can shift. For knowledge injection, consider full fine-tuning, LoRA with higher rank (128–256), or hybrid approaches (LoRA + continued pretraining).
“You only need to adapt attention”
Early papers adapted only q/v. For instruction tuning and reasoning tasks, adapting MLP layers (gate/up/down) often adds 1–3 points on benchmarks. The parameter cost is modest (~2x). Default to all linear layers unless memory forces you to restrict.
“Alpha doesn’t matter if you tune learning rate”
$\alpha$ controls initialization scale relative to base weights. With $\alpha = r$, initial LoRA output variance matches base layer output variance (assuming He init on base). If you set $\alpha=1$ and crank LR, you change the optimization trajectory — not equivalent. Treat $\alpha$ as part of the architecture, not a hyperparameter to absorb LR changes.
“LoRA works the same at any scale”
At 70B+, rank 16 on all layers adds ~40M params (0.06%) — still small but optimizer states grow. Gradient checkpointing becomes essential. Some teams use rank 8 or target only attention at 70B+ to keep memory manageable. The scaling laws for LoRA rank vs model size are not fully characterized; treat as empirical.
“Quantization and LoRA don’t mix”
They do. QLoRA (4-bit base + LoRA) is standard practice. The base loads in 4-bit NF4, LoRA adapters stay in BF16/FP32 for training. At inference, you can quantize the merged model to 4-bit or 8-bit. The PEFT library handles this automatically:
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3-70B",
quantization_config=bnb_config,
device_map="auto",
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)When to choose LoRA vs alternatives
| Scenario | Recommendation |
|---|---|
| Instruction tuning, style transfer, format compliance | LoRA (rank 16–32) |
| New language / massive domain knowledge | Full FT or continued pretraining + LoRA |
| Multi-tenant serving, rapid A/B | LoRA (unmerged, hot-swap) |
| Single-tenant, latency-critical | Merge LoRA, then quantize |
| Extreme parameter budget (<0.01%) | VeRA or LoRA rank 4–8 |
| Need exact full-FT parity | DoRA or full FT |
Practical checklist
- Start with rank 16, alpha 32, target all linear layers. This is the modern default that works across 7B–70B.
- Use BF16 compute, FP32 for LoRA params. Prevents gradient underflow in low-rank matrices.
- Set dropout 0.05–0.1 on LoRA path. Regularizes the adapter; base model stays deterministic.
- LR 1e-4 to 5e-4 for LoRA params. Base model LR = 0 (frozen). Cosine decay with 10% warmup.
- Gradient checkpointing on. Saves 30–40% activation memory.
- Eval every 500 steps. LoRA can overfit fast on small datasets; early stopping matters.
- Save adapter only.
trainer.save_model()with PEFT writesadapter_model.safetensors+adapter_config.json— ~25 MB for 7B. - Test merge before deploy.
merge_and_unload()can surface dtype mismatches (e.g., base in BF16, adapter in FP32).
Closing note
LoRA turned fine-tuning from a research-cluster operation into a standard engineering workflow. The technique is mature, the tooling (PEFT, TRL, Axolotl, Unsloth) is production-grade, and the trade-offs are well understood. If you’re customizing an LLM in 2024, LoRA should be your first attempt — not because it’s trendy, but because it solves the right constraints: memory, time, and operational flexibility.