LoRA vs QLoRA is the decision every team faces when fine-tuning large language models on constrained hardware. LoRA freezes the base model and trains low-rank adapters; QLoRA quantizes the base model to 4-bit before attaching those same adapters. The difference sounds small — quantization — but it shifts the hardware floor from 80 GB VRAM to 24 GB for a 7B model, and from multi-node clusters to a single 4090 for 70B. This post breaks down the trade-offs across memory, throughput, convergence, and ecosystem so you can pick without guessing.
What LoRA actually does
Low-Rank Adaptation injects trainable rank-decomposition matrices into the attention and MLP layers of a frozen transformer. For a weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA learns $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$ where $r \ll \min(d, k)$. The forward pass becomes:
# LoRA forward pass (simplified)
def lora_forward(x, W, A, B, scaling):
# W is frozen, A and B are trainable
return W @ x + scaling * (B @ (A @ x))The rank $r$ typically ranges from 8 to 64. At $r=16$ on a 7B model, you train roughly 0.1% of parameters — about 4.2M weights. The base model stays in FP16 or BF16. You need enough VRAM to hold the full model plus optimizer states for the adapters. For Llama-2-7B that’s ~14 GB for weights, ~6 GB for optimizer states, plus activations. A single 24 GB GPU handles it; 70B requires model parallelism or CPU offloading.
What QLoRA changes
QLoRA keeps the LoRA architecture but quantizes the frozen base model to 4-bit NormalFloat (NF4) with double quantization. The adapters still train in BF16. The quantization happens once, offline, using bitsandbytes:
# QLoRA model loading with bitsandbytes
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)The NF4 data type uses a quantile-based quantization scheme optimized for normally distributed weights. Double quantization compresses the quantization constants themselves, saving another ~0.5 bits/parameter. The result: Llama-2-7B fits in ~6 GB VRAM. Llama-2-70B fits in ~48 GB — doable on dual 3090s or a single H100.
Comparison table
| Dimension | LoRA (FP16/BF16 base) | QLoRA (4-bit NF4 base) |
|---|---|---|
| VRAM for 7B | ~20–24 GB | ~6–8 GB |
| VRAM for 70B | ~160 GB (multi-GPU) | ~48 GB (dual 3090 / single H100) |
| Training throughput | Baseline | 15–25% slower (dequant overhead) |
| Peak memory during backward | Higher (FP16 activations) | Lower (int4 weights, BF16 grads) |
| Final model quality | Reference | Within 0.5–1% on most benchmarks |
| Convergence stability | Stable | Slightly noisier gradients |
| Checkpoint size | Adapters only (~50 MB) | Adapters only (~50 MB) |
| Inference deployment | Merge + FP16/BF16 | Merge + quantize again, or serve 4-bit |
| Hardware floor (consumer) | RTX 3090/4090 (24 GB) | RTX 3060 12 GB (7B), 3090/4090 (70B) |
| Software maturity | PEFT, Axolotl, LLaMA-Factory | Same stack, requires bitsandbytes |
Memory and compute requirements
The memory savings come from storing the base model in 4-bit instead of 16-bit. A 7B parameter model drops from 14 GB to ~3.5 GB for weights. Optimizer states for LoRA adapters (AdamW: 2×FP32 per parameter) dominate the remaining budget. At $r=64$ on 7B, adapters have ~16M parameters → ~128 MB for optimizer states. Activations during training are the variable; gradient checkpointing reduces them by ~40% at a 20% throughput cost.
# Gradient checkpointing with PEFT
from peft import get_peft_model, LoraConfig, TaskType
peft_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=64,
lora_alpha=16,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, peft_config)
model.gradient_checkpointing_enable()
model.enable_input_require_grads() # Required for gradient checkpointingFor 70B models, LoRA without quantization needs 8×A100 80 GB or heavy CPU offloading via DeepSpeed ZeRO-3. QLoRA runs on 2×RTX 3090 (48 GB combined) with device_map="auto" splitting layers across GPUs. The trade-off: each forward/backward pass dequantizes 4-bit weights to BF16 on the fly. That dequantization kernel is memory-bandwidth bound, not compute bound, so throughput drops 15–25% versus FP16 LoRA on the same hardware.
Training throughput and latency
Benchmarking on a single A100 80 GB with Llama-2-7B, sequence length 2048, batch size 4:
- LoRA (BF16): ~3,200 tokens/sec
- QLoRA (NF4): ~2,600 tokens/sec
The gap narrows at larger batch sizes where compute utilization improves. On consumer GPUs with less memory bandwidth (RTX 4090: 1 TB/s vs A100: 1.5 TB/s), the relative penalty is similar. If you’re training for days, that 20% adds up. If you’re iterating on a single 4090, QLoRA is the only option that fits 70B.
One practical note: QLoRA requires bitsandbytes compiled for your CUDA version. On Windows, use the pre-built wheels from the official repo. On Linux, pip install bitsandbytes works if your toolchain matches. Mismatched versions manifest as symbol not found errors in libbitsandbytes_cuda.so — frustrating but fixable.
Model quality and convergence
The original QLoRA paper (Dettmers et al., 2023) showed Guanaco-7B (QLoRA on LLaMA-7B) matching ChatGPT-3.5 on Vicuna benchmarks. Subsequent work confirms: for instruction tuning and domain adaptation, QLoRA reaches within 0.5–1% of full LoRA on MMLU, GSM8K, and HumanEval. The gap widens slightly on tasks requiring precise numerical reasoning or long-context retrieval.
Why? Quantization noise. NF4 introduces ~0.5% weight error (cosine similarity). During training, gradients flow through dequantized weights, so the adapter learns to compensate. But the frozen base model’s representations are slightly degraded. For most downstream tasks — classification, summarization, code generation — this doesn’t matter. For high-stakes domains (medical, legal, financial), some teams prefer LoRA on FP16 to eliminate any quantization artifact risk.
Convergence curves look nearly identical for the first 50–80% of training. QLoRA sometimes shows slightly higher loss variance in the final 10–20%. Increasing lora_alpha (scaling) by 1.5–2× can stabilize it. A common QLoRA config that works well:
peft_config = LoraConfig(
r=64,
lora_alpha=32, # 2x rank, not 1x
lora_dropout=0.05,
bias="none",
target_modules="all-linear", # PEFT 0.10+ convenience
)Ecosystem and tooling
Both methods share the same PEFT (Parameter-Efficient Fine-Tuning) library. The difference is one import and a quantization config. Training frameworks — Axolotl, LLaMA-Factory, TRL, Unsloth — support both with a single flag.
# Axolotl config snippet for QLoRA
base_model: meta-llama/Llama-2-7b-hf
adapter: lora
load_in_4bit: true
bnb_4bit_quant_type: nf4
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_use_double_quant: true
lora_r: 64
lora_alpha: 32
lora_dropout: 0.05Unsloth deserves special mention: their optimized Triton kernels accelerate both LoRA and QLoRA by 2–3× on Hopper and Ada Lovelace architectures. They fuse the dequantization + matmul for QLoRA, recovering most of the throughput gap. If you’re on H100 or RTX 40-series, use Unsloth.
Merging adapters back into the base model works identically:
# Merge and save for inference
model = model.merge_and_unload()
model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")For QLoRA, the merged model is FP16/BF16. You can re-quantize it to 4-bit GPTQ or AWQ for deployment using AutoGPTQ or llama.cpp. The adapter weights themselves are tiny (~50 MB for 7B at r=64), so versioning and A/B testing multiple adapters is trivial.
Which to choose
Single GPU, 12–24 GB VRAM (RTX 3060/3080/3090/4090)
QLoRA only. LoRA won’t fit 7B on 12 GB, won’t fit 13B on 24 GB without aggressive offloading. QLoRA handles 7B on 12 GB, 13B on 24 GB, 34B on 24 GB with gradient checkpointing.
Dual 3090/4090 (48 GB combined)
QLoRA for 70B. LoRA needs 4×A100 or CPU offloading that makes training take weeks. QLoRA on dual 3090 trains 70B in 2–3 days.
Single A100/H100 80 GB
LoRA for 7B–13B, QLoRA for 34B–70B. On 80 GB you have headroom. LoRA gives 20% faster iteration. For 70B, QLoRA still fits; LoRA needs model parallelism.
Multi-node A100/H100 cluster
LoRA (or full fine-tuning). If you have the cluster, quantization overhead isn’t worth it. Full fine-tuning with ZeRO-3 or FSDP often beats LoRA on quality for 7B–13B at scale.
Regulated domains (medical, legal, finance) where quantization artifacts are unacceptable
LoRA on FP16/BF16. The 0.5% quality gap might matter. Document your reasoning.
Rapid experimentation, multiple adapter variants
QLoRA. Lower hardware cost means more parallel runs. Adapter checkpoints are identical size either way.
Production inference with n4n.ai or similar gateway
Either. Export merged FP16 for maximum compatibility, or merged 4-bit GPTQ/AWQ for cost. The gateway handles routing and fallback regardless of quantization.
Bottom line: Default to QLoRA. It unlocks model sizes that LoRA can’t reach on the same hardware, and the quality difference is negligible for almost every application. Only reach for FP16 LoRA when you have abundant VRAM, need maximum throughput, or operate in a domain where any quantization risk must be documented and justified.