How LoRA fine-tuning works, step by step

LoRA (Low-Rank Adaptation) has become the default approach for adapting large language models without full fine-tuning. If you’re asking how does LoRA fine-tuning work in practice, this guide walks through the complete pipeline — from preparing data to verifying the adapted model behaves as expected. We’ll use Hugging Face PEFT and the transformers library with a concrete example you can run on a single GPU.

Step 1: Understand the rank decomposition

LoRA freezes the base model weights and injects trainable rank-decomposition matrices into the attention layers. For a weight matrix W ∈ ℝ^(d×k), LoRA learns two smaller matrices A ∈ ℝ^(d×r) and B ∈ ℝ^(r×k) where r << min(d, k). The forward pass becomes:

h = Wx + (BA)x = Wx + ΔWx

Only A and B are updated during training. The rank r controls the expressivity-compute tradeoff. Typical values range from 8 to 64. Higher rank captures more complexity but increases memory and training time.

The scaling factor α (alpha) is typically set to 2 * r or r. The effective update is scaled by α / r. This keeps the magnitude of updates consistent across different rank choices.

Step 2: Prepare your environment and data

Install the required packages:

pip install torch transformers peft datasets accelerate bitsandbytes

For this example we’ll fine-tune microsoft/phi-2 (2.7B parameters) on a small instruction dataset. Phi-2 fits in 16GB VRAM with 4-bit quantization. Create a JSONL file train.jsonl with your training examples:

{"instruction": "Summarize the following text in one sentence.", "input": "The transformer architecture...", "output": "Transformers use self-attention for sequence modeling."}
{"instruction": "Write a Python function that computes fibonacci numbers.", "input": "", "output": "def fib(n):\n    a, b = 0, 1\n    for _ in range(n):\n        a, b = b, a + b\n    return a"}

Load and format the dataset:

from datasets import load_dataset

def format_example(example):
    if example["input"]:
        prompt = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n"
    else:
        prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n"
    return {"text": prompt + example["output"] + "<|endoftext|>"}

dataset = load_dataset("json", data_files="train.jsonl", split="train")
dataset = dataset.map(format_example, remove_columns=dataset.column_names)

Split into train/eval:

dataset = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset["train"]
eval_dataset = dataset["test"]

Step 3: Load the base model with quantization

QLoRA combines LoRA with 4-bit quantization to further reduce memory. Load the model in 4-bit using bitsandbytes:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "microsoft/phi-2"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

Verify the model loads and runs inference:

inputs = tokenizer("### Instruction:\nSay hello\n\n### Response:\n", return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

You should see a coherent response. If this fails, check CUDA availability and VRAM.

Step 4: Configure and apply LoRA

Define the LoRA configuration. Target the attention projection modules — for Phi-2 these are q_proj, k_proj, v_proj, dense (output projection), and optionally the MLP layers:

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "dense", "fc1", "fc2"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

Output should show roughly 0.5-1% trainable parameters:

trainable params: 1,572,864 || all params: 2,717,388,800 || trainable%: 0.0579%

The prepare_model_for_kbit_training call enables gradient checkpointing and prepares the model for 4-bit training. If you’re training without quantization, skip that line.

Step 5: Tokenize the dataset

Tokenize with a fixed max length. Truncate long sequences and pad to the max length in each batch:

def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=2048,
        padding=False,
        return_tensors=None,
    )

train_dataset = train_dataset.map(tokenize_function, batched=True, remove_columns=["text"])
eval_dataset = eval_dataset.map(tokenize_function, batched=True, remove_columns=["text"])

Use a data collator that handles dynamic padding:

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False,
    pad_to_multiple_of=8,
)

Step 6: Configure training arguments

Set up TrainingArguments for a single GPU run. Adjust per_device_train_batch_size and gradient_accumulation_steps to fit your VRAM:

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./phi2-lora-output",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=50,
    learning_rate=2e-4,
    fp16=False,
    bf16=True,
    logging_steps=10,
    evaluation_strategy="steps",
    eval_steps=50,
    save_steps=100,
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    report_to="none",
    remove_unused_columns=False,
    optim="paged_adamw_8bit",
)

Key points:

paged_adamw_8bit uses 8-bit optimizer states from bitsandbytes, saving additional memory
bf16=True requires Ampere or newer GPU (RTX 30-series, A100, H100). Use fp16=True on older GPUs
gradient_accumulation_steps=4 with batch_size=2 gives an effective batch size of 8

Step 7: Train the adapter

Create the Trainer and start training:

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=data_collator,
)

trainer.train()

Training on a single A10G (24GB) with these settings takes roughly 15-20 minutes for 3 epochs on ~1000 examples. Monitor the eval loss — it should decrease steadily. If it plateaus or increases, reduce the learning rate.

Step 8: Save and merge the adapter

After training, save the LoRA adapter weights separately. This is the portable artifact you can share or version:

model.save_pretrained("./phi2-lora-adapter")
tokenizer.save_pretrained("./phi2-lora-adapter")

The adapter directory contains adapter_model.safetensors (the A and B matrices) and adapter_config.json.

To merge the adapter into the base model for deployment (so you don’t need PEFT at inference time):

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

merged_model = PeftModel.from_pretrained(base_model, "./phi2-lora-adapter")
merged_model = merged_model.merge_and_unload()

merged_model.save_pretrained("./phi2-lora-merged")
tokenizer.save_pretrained("./phi2-lora-merged")

The merged model is a standard transformers model — load it without PEFT:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("./phi2-lora-merged", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("./phi2-lora-merged")

Step 9: Verify the fine-tuned model

Run a systematic evaluation. Create a test set with held-out instructions:

test_prompts = [
    "### Instruction:\nExplain what a closure is in Python.\n\n### Response:\n",
    "### Instruction:\nWrite a SQL query to find the second highest salary.\n\n### Response:\n",
    "### Instruction:\nSummarize: The attention mechanism allows models to weigh input tokens differently.\n\n### Response:\n",
]

for prompt in test_prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=150,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    print(f"Prompt: {prompt[:80]}...")
    print(f"Response: {response}\n{'-'*60}")

Compare outputs against the base model. The fine-tuned model should follow the instruction format more reliably and show domain-specific knowledge from your training data.

For quantitative verification, compute perplexity on a held-out validation set:

import math

def compute_perplexity(model, dataset, tokenizer, max_samples=100):
    model.eval()
    total_loss = 0
    total_tokens = 0
    with torch.no_grad():
        for i, example in enumerate(dataset.select(range(min(max_samples, len(dataset))))):
            inputs = {k: torch.tensor(v).unsqueeze(0).to("cuda") for k, v in example.items()}
            outputs = model(**inputs, labels=inputs["input_ids"])
            loss = outputs.loss
            total_loss += loss.item() * inputs["input_ids"].numel()
            total_tokens += inputs["input_ids"].numel()
    avg_loss = total_loss / total_tokens
    return math.exp(avg_loss)

print(f"Perplexity: {compute_perplexity(model, eval_dataset, tokenizer):.2f}")

Lower perplexity than the base model on your domain data confirms the adaptation worked.

Step 10: Deploy or serve the model

The merged model can be served with any OpenAI-compatible server. For example, with vllm:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model ./phi2-lora-merged \
    --dtype bfloat16 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.9

Or load it directly in your application:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="./phi2-lora-merged",
    messages=[
        {"role": "user", "content": "Write a haiku about LoRA fine-tuning."}
    ],
    temperature=0.7,
)
print(response.choices[0].message.content)

If you’re running an inference gateway that handles routing across multiple adapted models, you can register each LoRA adapter as a separate model variant pointing to the same base — this avoids loading duplicate base weights. n4n.ai supports this pattern by letting you specify the base model and adapter path per route.

Common failure modes and fixes

Symptom	Likely cause	Fix
OOM on forward pass	Batch size too large, max_length too long	Reduce `per_device_train_batch_size`, `max_length`, or increase `gradient_accumulation_steps`
Loss goes to NaN	Learning rate too high, unstable quantization	Lower `learning_rate` to 1e-4, ensure `bf16`/`fp16` matches GPU capability
No improvement in eval loss	Rank too low, insufficient data, wrong target modules	Increase `r` to 32/64, add more data, verify `target_modules` match model architecture
Generated text repeats or degrades	Missing `pad_token`, wrong `padding_side`	Set `tokenizer.pad_token = tokenizer.eos_token`, `padding_side = "right"`
Merged model performs worse	`merge_and_unload` called on quantized model	Merge on full-precision base, or use `merge_and_unload(progressbar=True)` and verify weights

Choosing rank and alpha

Start with r=16, alpha=32. If eval loss plateaus early, increase rank. If training is unstable, decrease learning rate before increasing rank. For simple style/formatting tasks, r=8 often suffices. For knowledge injection or complex reasoning, r=32-64 works better. The alpha parameter scales the LoRA output — think of it as a learning rate multiplier for the adapter. Keeping alpha = 2 * r is a safe default.

When to use full fine-tuning instead

LoRA excels when:

You have limited compute (single GPU, consumer hardware)
The task is stylistic, formatting, or narrow domain adaptation
You need to maintain multiple task-specific adapters

Consider full fine-tuning when:

You’re changing the model’s fundamental capabilities (new languages, code →languages, massive knowledge injection)
You have multi-GPU clusters and the compute budget
The base model is small enough (< 7B) that full fine-tuning is tractable

You now have a complete, runnable LoRA fine-tuning pipeline. The adapter weights are portable, the merged model deploys anywhere, and the verification steps give you confidence the adaptation actually improved the model on your task. Start small, iterate on data quality, and scale rank only when the eval metrics demand it.