n4nAI
Open console →
GlossaryTutorialLoRA, QLoRA & Parameter-Efficient Fine-Tuning

Fine-tuning Llama 3 with LoRA: what you need to know

A hands-on tutorial for fine-tuning Llama 3 with LoRA, covering prerequisites, dataset prep, training loops, and evaluation with runnable code.

n4n Team3 min read635 words

Audio narration

Coming soon — every post will get a voice note here.

Fine-tuning Llama 3 with LoRA lets you adapt a 7B or 8B parameter model on a single GPU with 24 GB VRAM. The technique freezes the base weights and trains low-rank adapters — typically 0.1–1% of the parameters — so you get task-specific performance without the compute budget of full fine-tuning. This tutorial walks through a complete run on an A100 or H100, from environment setup to a merged checkpoint you can serve.

Prerequisites

You need a Linux box with an NVIDIA GPU (24 GB VRAM minimum for 7B/8B at 4-bit), CUDA 12.1+, and Python 3.10 or 3.11. Install the stack in a fresh venv:

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install torch==2.3.0 --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.41.0 peft==0.11.1 trl==0.9.4 accelerate==0.30.1 bitsandbytes==0.43.1 datasets==2.19.0

Verify bitsandbytes compiled correctly:

python -c "import bitsandbytes as bnb; print(bnb.__version__)"

Expected output:

0.43.1

If you see a CUDA version mismatch, reinstall bitsandbytes from source against your CUDA toolkit.

Model and tokenizer

Llama 3 uses a different tokenizer than Llama 2 — 128k vocabulary, byte-level BPE, no added space prefix. Load the 8B Instruct model in 4-bit NF4 with double quantization:

# load_model.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model.config.use_cache = False
model.config.pretraining_tp = 1

print("Model loaded:", model.device)
print("Trainable params:", sum(p.numel() for p in model.parameters() if p.requires_grad))

Run it:

python load_model.py

Expected output (approximate):

Model loaded: cuda:0
Trainable params: 0

All parameters are frozen. The LoRA adapters will be the only trainable weights.

LoRA configuration

Target the attention projection modules. For Llama 3, the module names are q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Rank 16 with alpha 32 is a solid default; scale rank up for more capacity, down for speed.

# lora_config.py
from peft import LoraConfig, TaskType, get_peft_model

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

def print_trainable_parameters(model):
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    total = sum(p.numel() for p in model.parameters())
    print(f"Trainable: {trainable:,} | Total: {total:,} | %: {100 * trainable / total:.4f}")

# Usage after loading model:
# model = get_peft_model(model, lora_config)
# print_trainable_parameters(model)

Expected trainable parameter count for 8B at rank 16:

Trainable: 8,388,608 | Total: 8,030,261,248 | %: 0.1045

Dataset preparation

Use a chat-format dataset. The trl library expects a messages column with a list of {"role": "user|assistant|system", "content": "..."} dicts. Here’s a minimal example using the databricks/databricks-dolly-15k dataset, filtered to English and formatted:

# prepare_data.py
from datasets import load_dataset
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
tokenizer.pad_token = tokenizer.eos_token

def format_chat(example):
    prompt = tokenizer.apply_chat_template(
        example["messages"],
        tokenize=False,
        add_generation_prompt=False,
    )
    return {"text": prompt}

ds = load_dataset("databricks/databricks-dolly-15k", split="train")

# Convert dolly to chat format
def to_chat(example):
    return {
        "messages": [
            {"role": "user", "content": example["instruction"] + ("\n" + example["context"] if example["context"] else "")},
            {"role": "assistant", "content": example["response"]},
        ]
    }

ds = ds.map(to_chat, remove_columns=ds.column_names)
ds = ds.map(format_chat, remove_columns=["messages"])

# Train/val split
ds = ds.train_test_split(test_size=0.05, seed=42)
ds["train"].to_json("train.jsonl", orient="records", lines=True)
ds["test"].to_json("val.jsonl", orient="records", lines=True)

print(f"Train: {len(ds['train'])}, Val: {len(ds['test'])}")

Run it:

python prepare_data.py

Expected output:

Train: 14250, Val: 750

Training with SFTTrainer

trl.SFTTrainer handles packing, masking, and logging. Key settings: max_seq_length=4096, packing=True for throughput, gradient_accumulation_steps=4 to simulate batch size 16 on a single GPU.

# train.py
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
from datasets import load_dataset

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
)
model.config.use_cache = False
model.config.pretraining_tp = 1

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

train_dataset = load_dataset("json", data_files="train.jsonl", split="train")
val_dataset = load_dataset("json", data_files="val.jsonl", split="train")

training_args = TrainingArguments(
    output_dir="./llama3-8b-lora-dolly",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    evaluation_strategy="steps",
    eval_steps=100,
    save_steps=200,
    save_total_limit=3,
    bf16=True,
    tf32=True,
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",
    report_to="tensorboard",
    seed=42,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    dataset_text_field="text",
    max_seq_length=4096,
    packing=True,
    tokenizer=tokenizer,
)

trainer.train()
trainer.save_model("./llama3-8b-lora-dolly/final")

Launch training:

python train.py

Expected log snippet (first few steps):

{'loss': 2.847, 'learning_rate': 6.0e-06, 'epoch': 0.01, 'step': 10}
{'loss': 2.612, 'learning_rate': 1.2e-05, 'epoch': 0.02, 'step': 20}
{'eval_loss': 2.103, 'epoch': 0.05, 'step': 100}

Training 3 epochs on 14k samples takes roughly 2.5 hours on an A100 80 GB. On 24 GB VRAM, reduce max_seq_length to 2048 and per_device_train_batch_size to 2.

Merge and export

After training, merge the LoRA weights into the base model for deployment. This produces a standard safetensors checkpoint you can serve with vLLM, TGI, or any OpenAI-compatible server.

# merge.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
adapter_path = "./llama3-8b-lora-dolly/final"
output_path = "./llama3-8b-dolly-merged"

base = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

model = PeftModel.from_pretrained(base, adapter_path)
model = model.merge_and_unload()

model.save_pretrained(output_path, safe_serialization=True)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
tokenizer.save_pretrained(output_path)

print(f"Merged model saved to {output_path}")

Run it:

python merge.py

Expected output:

Merged model saved to ./llama3-8b-dolly-merged

Verify the merged checkpoint loads cleanly:

python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained('./llama3-8b-dolly-merged', torch_dtype=torch.bfloat16, device_map='auto')
t = AutoTokenizer.from_pretrained('./llama3-8b-dolly-merged')
print('Merged model params:', sum(p.numel() for p in m.parameters()))
"

Quick inference test

Sanity-check the merged model with a few prompts:

# test_inference.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "./llama3-8b-dolly-merged"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompts = [
    "Explain LoRA in two sentences.",
    "Write a Python function that computes the Fibonacci sequence iteratively.",
    "What is the capital of Australia?",
]

for p in prompts:
    messages = [{"role": "user", "content": p}]
    input_ids = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)

    out = model.generate(
        input_ids,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )
    response = tokenizer.decode(out[0][input_ids.shape[-1]:], skip_special_tokens=True)
    print(f"User: {p}\nAssistant: {response}\n{'-'*60}")

Sample output:

User: Explain LoRA in two sentences.
Assistant: LoRA (Low-Rank Adaptation) freezes a pretrained model's weights and injects trainable low-rank matrices into the attention layers, reducing trainable parameters by orders of magnitude. This enables efficient fine-tuning on consumer hardware while preserving the base model's knowledge.
------------------------------------------------------------
User: Write a Python function that computes the Fibonacci sequence iteratively.
Assistant: def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a
------------------------------------------------------------
User: What is the capital of Australia?
Assistant: The capital of Australia is Canberra.
------------------------------------------------------------

Common pitfalls

OOM on 24 GB VRAM. Drop max_seq_length to 2048, per_device_train_batch_size to 1, and increase gradient_accumulation_steps to 16. Enable gradient_checkpointing=True (already set) and optim="paged_adamw_8bit".

Loss spikes or NaN. Lower learning_rate to 1e-4, increase warmup_ratio to 0.05, and ensure bf16=True with torch_dtype=torch.bfloat16. Flash Attention 2 requires attn_implementation="flash_attention_2" on both load and merge.

Tokenizer mismatch at inference. Always save the tokenizer alongside the merged model. Llama 3’s tokenizer adds no leading space; if you see garbled first tokens, verify tokenizer.padding_side = "right" and pad_token = eos_token were set consistently.

LoRA rank too low for complex tasks. If eval loss plateaus high, bump r to 32 or 64 and lora_alpha to 64 or 128. Trainable params scale linearly with rank; 8B at r=64 is ~0.4% trainable.

Serving the merged model

The merged checkpoint is a standard Hugging Face model. Serve it with vLLM for production throughput:

pip install vllm==0.5.0
python -m vllm.entrypoints.openai.api_server \
  --model ./llama3-8b-dolly-merged \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9

This exposes an OpenAI-compatible endpoint at http://localhost:8000/v1. You can route traffic through an inference gateway that handles fallback and usage metering — n4n.ai does this with a single endpoint addressing 240+ models and automatic provider fallback when a backend is rate-limited.

Next steps

  • QLoRA 4-bit training: The code above already uses 4-bit NF4 base weights. For true QLoRA, add peft.prepare_model_for_kbit_training(model) before get_peft_model.
  • DPO/ORPO alignment: After SFT, run Direct Preference Optimization on a preference dataset to improve instruction following.
  • Longer context: Llama 3 supports 8k natively; for 128k, use YaRN or LongRoPE scaling and extend max_seq_length gradually with continued pretraining.
  • Multi-GPU: Switch device_map="auto" to accelerate launch --config_file fsdp_config.yaml train.py for FSDP or DeepSpeed ZeRO-3.

The LoRA adapters you trained are portable — copy adapter_model.safetensors and adapter_config.json to any Llama 3 8B base and PeftModel.from_pretrained will load them. This separation of base model and task adapters is the operational pattern that scales.

Tagslorallama-3fine-tuningpeft

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
GlossaryHow LoRA fine-tuning works, step by stepA practical step-by-step guide to LoRA fine-tuning with runnable code, covering setup, training, and verification for engineers building with LLMs.GlossaryLoRA vs full fine-tuning: costs and tradeoffsA practical head-to-head comparison of LoRA and full fine-tuning across cost, performance, and operational tradeoffs for engineers choosing a fine-tuning strategy.GlossaryLoRA vs QLoRA: what's the difference?A practitioner's head-to-head comparison of LoRA and QLoRA across memory, compute, quality, and tooling — with a clear verdict for each use case.GlossaryWhat is LoRA? Low-rank adaptation explainedLoRA (Low-Rank Adaptation) explained for engineers — how it works, why it matters for fine-tuning LLMs, and a concrete PyTorch example.GlossaryWhat is PEFT? Parameter-efficient fine-tuning explainedPEFT explained for engineers — what it is, how LoRA and QLoRA work, when to use each, and the trade-offs you'll hit in production.GlossaryQLoRA explained: fine-tuning in 4-bit precisionQLoRA combines 4-bit quantization with LoRA adapters to fine-tune large language models on consumer GPUs without quality loss.

All lora, qlora & parameter-efficient fine-tuning posts →