What is PEFT? Parameter-efficient fine-tuning explained

Parameter-efficient fine-tuning (PEFT) is a family of techniques that adapt pre-trained language models by updating only a small fraction of their parameters — typically under 1% — while freezing the rest. Instead of backpropagating through billions of weights, PEFT methods inject trainable adapters or low-rank matrices into the model architecture, dramatically reducing memory and compute requirements. The result: you can fine-tune a 70B parameter model on a single GPU with 24GB VRAM, something full fine-tuning would require a multi-node cluster to accomplish.

Why full fine-tuning breaks at scale

Full fine-tuning updates every weight in the model. For a 7B parameter model at fp16, that’s 14GB just for weights, plus another 14GB for gradients, plus optimizer states (AdamW keeps two momentum buffers per parameter — another 56GB). You’re looking at 80GB+ GPU memory before you even load a batch. Multi-GPU setups with ZeRO-3 or FSDP can shard this, but the engineering overhead is real: distributed training fragility, communication overhead, checkpoint management, and the operational cost of keeping a cluster healthy.

PEFT sidesteps this by freezing the base model and training only a tiny set of added parameters. The base model stays in fp16 or even int4/8 quantization; only the adapter weights need fp32 for optimizer states. A 7B model with LoRA at rank 64 adds roughly 4M trainable parameters — about 0.06% of the base — dropping optimizer memory to megabytes.

How LoRA actually works

Low-Rank Adaptation (LoRA) is the most widely adopted PEFT method. The core insight: weight updates during fine-tuning have low intrinsic rank. Instead of learning a full ΔW ∈ ℝ^(d×k), LoRA factorizes it into two smaller matrices A ∈ ℝ^(d×r) and B ∈ ℝ^(r×k) where r ≪ min(d, k). The forward pass becomes:

# Original: y = x @ W
# LoRA: y = x @ W + x @ (A @ B) * scaling
# where scaling = alpha / r

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, r=64, alpha=16, dropout=0.0):
        super().__init__()
        self.r = r
        self.alpha = alpha
        self.scaling = alpha / r
        
        # Frozen base weight
        self.weight = nn.Parameter(torch.empty(out_features, in_features))
        self.weight.requires_grad = False
        
        # Trainable low-rank matrices
        self.lora_A = nn.Parameter(torch.empty(r, in_features))
        self.lora_B = nn.Parameter(torch.zeros(out_features, r))
        self.dropout = nn.Dropout(dropout)
        
        # Init: A ~ N(0, 1/sqrt(r)), B = 0 so initial output = base model
        nn.init.normal_(self.lora_A, std=1 / r**0.5)
    
    def forward(self, x):
        base = F.linear(x, self.weight)
        lora = (self.dropout(x) @ self.lora_A.T @ self.lora_B.T) * self.scaling
        return base + lora

The rank r controls the expressiveness-compression tradeoff. Typical values: 8, 16, 32, 64, 128. Higher rank = more capacity but more parameters. The alpha hyperparameter scales the LoRA output; common practice sets alpha = 2 * r or alpha = r, making scaling = 2 or 1. During inference, you can merge LoRA weights into the base model (W_merged = W + B @ A * scaling) for zero latency overhead.

Target modules: where to inject LoRA

You don’t apply LoRA to every linear layer. The standard recipe targets attention projection matrices: q_proj, k_proj, v_proj, o_proj in each transformer block. Some configurations also include gate_proj, up_proj, down_proj in the MLP. Targeting only attention is cheaper (fewer parameters) and often sufficient; targeting all linear layers (“full LoRA”) can improve quality on complex tasks but increases trainable params 3-4x.

# peft.LoraConfig example
from peft import LoraConfig, TaskType

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=64,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none",
)

QLoRA: quantization + LoRA

QLoRA (Quantized LoRA) pushes memory efficiency further by loading the base model in 4-bit NormalFloat (NF4) quantization with double quantization, then applying LoRA adapters in fp16/bf16. The base model stays frozen in 4-bit; only LoRA weights are dequantized during forward/backward. This fits a 70B model on a single 48GB GPU (or 65B on 24GB with gradient checkpointing).

# bitsandbytes + peft QLoRA setup
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training, get_peft_model

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, config)

Key QLoRA details that matter in practice:

• NF4 quantization uses a data-type optimized for normally distributed weights, outperforming int4 symmetric quantization.
• Double quantization quantizes the quantization constants themselves, saving ~0.5 bits/parameter.
• Paged optimizers (via bitsandbytes) offload optimizer states to CPU when GPU memory pressure spikes, preventing OOM on long sequences.
• Gradient checkpointing is essential — recomputes activations during backward pass, trading 20-30% compute for 2-3x activation memory savings.

Other PEFT methods worth knowing

Adapter modules (Houlsby/AdapterHub)

Inserts small bottleneck MLPs between transformer sub-layers. Original paper: two-layer MLP with residual connection, hidden dim 64. More parameters than LoRA for equivalent capacity, but architecturally cleaner — no weight merging needed, easier to stack multiple adapters. Less common in open-source LLM fine-tuning now, but still used in multi-task setups.

Prefix tuning / P-tuning / Prompt tuning

Prepends trainable continuous vectors (virtual tokens) to the input or each layer’s hidden states. Only the prefix embeddings train; the model sees them as extra context. Extremely parameter-efficient (few thousand params), but typically underperforms LoRA on knowledge-intensive tasks. Useful when you need many task-specific adaptations sharing one base model — swap prefixes at inference.

IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations)

Scales activations with learned vectors instead of adding low-rank updates. Even fewer parameters than LoRA (two vectors per layer), competitive on some benchmarks. Less battle-tested in production.

LoRA variants

• DoRA (Weight-Decomposed Low-Rank Adaptation): Separates magnitude and direction updates, matches full fine-tuning performance with LoRA parameter count. Slightly more complex implementation.
• VeRA (Vector-based Random Matrix Adaptation): Freezes A and B as random orthogonal matrices, trains only scaling vectors. Drops trainable params 10x vs LoRA. Quality holds on some tasks, degrades on others.
• LoRA+ / LoRA-GA / LoRA-Drop: Various learning rate scheduling and gradient manipulation tweaks. Marginal gains, more hyperparameters.

When to use which method

Concrete example: fine-tuning Llama-3-8B for SQL generation

# Hardware: 1× H100 80GB (or A100 80GB)
# Dataset: 50k text-to-SQL pairs, avg 2k tokens
# Target: match or beat GPT-3.5-Turbo on Spider benchmark

# train.py
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    TrainingArguments, Trainer, DataCollatorForSeq2Seq
)
from peft import LoraConfig, get_peft_model

model_id = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

lora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,291,456 || all params: 8,030,261,248 || trainable%: 0.0783%

def format_example(example):
    prompt = f"""### Instruction:
Write SQLite SQL for: {example['question']}

### Context:
{example['context']}

### Response:
{example['sql']}"""
    return {"text": prompt}

dataset = load_dataset("b-mc2/sql-create-context", split="train")
dataset = dataset.map(format_example).shuffle(seed=42).select(range(50000))

def tokenize(example):
    return tokenizer(
        example["text"],
        truncation=True,
        max_length=4096,
        padding=False,
    )

dataset = dataset.map(tokenize, remove_columns=dataset.column_names)

training_args = TrainingArguments(
    output_dir="llama3-8b-sql-lora",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    optim="adamw_torch_fused",
    report_to="wandb",
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=DataCollatorForSeq2Seq(tokenizer, pad_to_multiple_of=8),
)

trainer.train()

# Merge and save for inference
model.merge_and_unload()
model.save_pretrained("llama3-8b-sql-merged")
tokenizer.save_pretrained("llama3-8b-sql-merged")

Training time: ~6 hours on H100. VRAM peak: ~55GB. The merged model runs at base model latency — no LoRA overhead at inference.

Common misconceptions

“LoRA always matches full fine-tuning quality”

False. On knowledge-intensive tasks (new domain facts, new languages, complex reasoning), full fine-tuning still wins. LoRA excels at style transfer, format adherence, and skill composition — tasks where the base model already has the relevant knowledge. The gap narrows with higher rank (128-256) and targeting all linear layers, but memory scales accordingly. QLoRA adds a small quantization degradation on top.

“You can merge LoRA weights and keep training”

No. Merging (W = W + B @ A * scaling) bakes the adapter into the base weights. You cannot “un-merge” cleanly because the optimizer states (Adam momentum) are tied to the original parameterization. If you need to continue training, keep the adapter separate. Merge only for final inference deployment.

“Higher rank is always better”

Diminishing returns hit hard past r=128 for most tasks. Higher rank increases overfitting risk on small datasets and slows training (more matmul FLOPs). Start at r=32 or 64, scale up only if validation loss plateaus and you have data to support it.

“QLoRA 4-bit is free quality”

4-bit quantization degrades perplexity ~0.1-0.3 points vs fp16 base. On downstream tasks, the gap is often negligible after fine-tuning, but not always. For high-stakes production (code generation, medical, legal), validate against fp16 LoRA on your eval set. The 2x memory savings are real; the quality cost is task-dependent.

“PEFT means you need less data”

PEFT reduces parameter count, not data requirements. You still need enough examples to learn the task — typically 1k-10k for format/style tasks, 10k-100k+ for knowledge injection. With too little data, LoRA overfits faster than full fine-tuning because the low-rank constraint forces the model to compress the update into fewer degrees of freedom.

“LoRA adapters compose trivially”

You can add multiple LoRA adapters (W + Σ B_i @ A_i * scaling_i), but they interfere. Sequential training (train adapter A, freeze, train adapter B) works better than joint training for distinct tasks. For inference-time composition, merge each adapter separately, then average merged weights — or use methods like TIES-Merging, DARE, or Model Breadcrumbs. Naive weight averaging often degrades both tasks.

Production considerations

Checkpointing and versioning

Save only adapter weights (adapter_model.safetensors, ~50MB for 7B r=64) plus adapter_config.json. The base model stays unchanged — reference it by hash or version tag. This makes CI/CD trivial: build base model once, ship adapter artifacts per task.

// adapter_config.json
{
  "peft_type": "LORA",
  "r": 64,
  "lora_alpha": 16,
  "lora_dropout": 0.05,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"],
  "bias": "none",
  "task_type": "CAUSAL_LM"
}

Serving merged vs unmerged

• Merged: Zero latency overhead, standard vLLM/TGI/TensorRT-LLM deployment. One model artifact per task.
• Unmerged (dynamic LoRA): Load base once, swap adapters per request. vLLM and TGI support this via lora_request API. Adds ~5-15% latency per request (extra matmul), but serves 100s of tasks from one base model replica. Memory: base + sum of active adapters.

# vLLM dynamic LoRA example
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model="meta-llama/Meta-Llama-3-8B", enable_lora=True, max_lora_rank=64)

sql_lora = LoRARequest("sql", 1, "llama3-8b-sql-lora")
python_lora = LoRARequest("python", 2, "llama3-8b-python-lora")

outputs = llm.generate(
    ["Write SQL for...", "Write Python for..."],
    sampling_params=SamplingParams(max_tokens=512),
    lora_request=[sql_lora, python_lora],
)

Monitoring adapter drift

Track training loss, eval loss, and base model perplexity on a held-out domain set during training. If base perplexity rises, the adapter is damaging the model’s general capabilities — reduce learning rate, lower rank, or add regularization. This is the “catastrophic forgetting” signal for PEFT.

Summary decision tree

1. GPU memory ≥ 2× model size in fp16 → Full fine-tuning (or DoRA for near-PEFT params)
2. GPU memory ≥ model size in fp16 → LoRA bf16, r=64-128, target all linear layers
3. GPU memory < model size in fp16 → QLoRA 4-bit NF4, r=64, gradient checkpointing, paged optimizer
4. Many tasks, one base, inference-time swap → Dynamic LoRA serving or prefix tuning
5. Extreme parameter budget (edge/mobile) → VeRA or IA³, accept quality tradeoff

PEFT isn’t a free lunch — it’s a precise engineering tradeoff. The methods are mature, the tooling (PEFT, bitsandbytes, Hugging Face Trainer, vLLM) is production-grade, and the operational benefits are real. But you still need to understand what you’re giving up: some quality on knowledge-heavy tasks, hyperparameter sensitivity around rank and learning rate, and the discipline to validate merged vs unmerged behavior before shipping.