Fine-tuning Llama 3: what you need to know

Fine-tuning Llama 3 isn’t magic, but it’s easy to waste GPU hours on the wrong knobs. This guide walks through the decisions that actually move the needle: data quality over quantity, LoRA ranks that fit your VRAM, and evaluation that catches regression before production. You’ll leave with a reproducible recipe and a mental model for when fine-tuning beats prompting, RAG, or a larger base model.

Start with the question you’re actually trying to answer

Before you spin up a single GPU, write down the failure mode of your current approach. Are you fighting format adherence? Domain vocabulary? Style and tone? A specific reasoning pattern? Each maps to a different data strategy and a different risk profile.

If the problem is “the model doesn’t know our internal acronyms,” fine-tuning is overkill — use RAG or a system prompt with a glossary. If the problem is “the model hallucinates our API signatures,” you need supervised fine-tuning (SFT) on verified examples. If it’s “the model refuses valid requests because of overzealous safety alignment,” you need preference optimization (DPO/ORPO) on your policy, not just more SFT.

Pitfall: Teams often fine-tune for knowledge injection. Don’t. Llama 3’s parametric knowledge is frozen at training cutoff. Fine-tuning teaches behavior, not facts. For facts, use retrieval.

Choose your model variant and quantization

Llama 3 ships as 8B and 70B parameter models, each in base and instruct flavors. For most teams, the choice is practical:

Variant	VRAM (BF16)	VRAM (4-bit QLoRA)	Use case
8B Instruct	~16 GB	~6 GB	Default starting point; fits single 24 GB GPU
70B Instruct	~140 GB	~48 GB	Need stronger reasoning; requires multi-GPU or A100 80 GB
8B Base	~16 GB	~6 GB	Continued pre-training or heavy domain adaptation
70B Base	~140 GB	~48 GB	Rarely justified unless you have massive compute budget

Start with Llama-3-8B-Instruct quantized to 4-bit NF4 via QLoRA. It fits on a single consumer GPU (RTX 3090/4090, 24 GB) with room for batch size 2–4 and gradient accumulation. The instruct version already understands chat formatting, tool calling, and basic safety — you’re teaching your task, not chat basics.

# bitsandbytes 4-bit NF4 config — copy-paste ready
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

Tradeoff: 4-bit QLoRA adds ~0.5–1% perplexity vs. BF16 full fine-tune, but reduces VRAM 4×. For 8B, that’s the difference between “runs on one GPU” and “needs two.” Take the hit.

Curate data like it’s production code

Your dataset is the model. A 1,000-example dataset of verified, diverse, correctly formatted conversations beats 50,000 scraped examples with hallucinations, truncated turns, and format drift.

Format: chat template compliance

{
  "messages": [
    {"role": "system", "content": "You are a precise API documentation assistant."},
    {"role": "user", "content": "How do I authenticate to the payments endpoint?"},
    {"role": "assistant", "content": "Use the `Authorization: Bearer <token>` header. Token format: `sk_live_...` or `sk_test_...`."}
  ]
}

Apply the tokenizer’s apply_chat_template with tokenize=False to verify rendering before training. A single missing <|eot_id|> cascades into generations that never stop.

Quality filters worth automating

def filter_example(example, tokenizer, max_tokens=4096):
    # 1. Length guard
    rendered = tokenizer.apply_chat_template(
        example["messages"], tokenize=False
    )
    if len(tokenizer(rendered).input_ids) > max_tokens:
        return False
    
    # 2. Role alternation (no consecutive user/user or assistant/assistant)
    roles = [m["role"] for m in example["messages"]]
    if any(roles[i] == roles[i+1] for i in range(len(roles)-1)):
        return False
    
    # 3. Assistant message present (SFT needs targets)
    if roles[-1] != "assistant":
        return False
    
    # 4. Heuristic: reject if assistant response < 10 chars or > 2000 chars
    assistant_text = example["messages"][-1]["content"]
    if len(assistant_text) < 10 or len(assistant_text) > 2000:
        return False
    
    return True

Diversity over volume

Cover the distribution of your production inputs: short queries, multi-turn context, edge cases, adversarial prompts, different personas. If 80% of your traffic is “summarize this ticket,” 80% of your data should be summarization — but the remaining 20% must include the weird stuff or the model overfits the happy path.

Pitfall: Copy-pasting the same system prompt into every example. Vary it. Production system prompts change; the model should be robust to that.

LoRA configuration: the knobs that matter

For Llama 3 8B at 4-bit, this config is a strong default:

from peft import LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                    # rank — 8, 16, 32, 64
    lora_alpha=32,           # scaling = alpha / r
    lora_dropout=0.05,       # dropout on LoRA adapters
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",  # attention
        "gate_proj", "up_proj", "down_proj",     # MLP
    ],
    bias="none",
    modules_to_save=None,    # set to ["embed_tokens", "lm_head"] if adding tokens
)

Rank (r) and alpha: the capacity tradeoff

r=8, alpha=16 — minimal capacity, fastest, lowest overfit risk. Good for style/format tuning.
r=16, alpha=32 — sweet spot for most SFT tasks. ~0.5% of base params trained.
r=32, alpha=64 — more capacity for complex reasoning or multi-task. Watch for overfit on small datasets.
r=64, alpha=128 — approaching full fine-tune capacity. Needs more data, more regularization.

Rule of thumb: alpha = 2 * r keeps effective learning rate stable across ranks. Don’t treat alpha as a separate hyperparameter.

Target modules: don’t skip the MLP

Early LoRA papers targeted only attention. For Llama 3, include the MLP projections (gate_proj, up_proj, down_proj). The SwiGLU feed-forward layers carry significant task-specific adaptation. Skipping them saves ~30% adapter params but costs 1–2 points on benchmarks.

Modules_to_save: when you need it

If you add special tokens (new control codes, domain-specific markers), set modules_to_save=["embed_tokens", "lm_head"] so the new embeddings train. Otherwise leave it None — training the full vocab head is wasteful and destabilizes generation.

Training loop: stability over speed

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./llama3-8b-lora",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,   # effective batch = 8
    num_train_epochs=3,
    learning_rate=2e-4,              # 1e-4 to 5e-4 for LoRA
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    max_grad_norm=1.0,
    bf16=True,                       # requires Ampere+ (RTX 30-series, A100, H100)
    tf32=True,
    gradient_checkpointing=True,     # saves ~30% VRAM, ~20% slower
    optim="paged_adamw_8bit",        # 8-bit optimizer states
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    report_to="tensorboard",
    seed=42,
    data_seed=42,
)

Learning rate: the most sensitive knob

Too high (5e-4+): Loss spikes, generation degrades to gibberish within 500 steps.
Too low (5e-5): Underfits, needs 2× epochs to converge.
2e-4 works for 90% of SFT on Llama 3 8B with r=16. If using r=64, drop to 1e-4.

Epochs vs. steps: stop early

Three epochs is a ceiling, not a target. Monitor eval loss every 50–100 steps. Stop when it plateaus or ticks up. Most 1k–5k example datasets converge in 1–2 epochs. More epochs = overfit to training format, not your task.

Pitfall: Training until training loss hits zero. That’s memorization, not learning. You want the gap between train and eval loss to stay small.

Gradient checkpointing + paged optimizer = single GPU viability

gradient_checkpointing=True recomputes activations on backward pass. optim="paged_adamw_8bit" offloads optimizer states to CPU. Together they make 8B QLoRA fit in 16–18 GB VRAM with batch size 2. Without both, you OOM.

Evaluation: don’t ship on vibes

You need three evaluation layers, and “it looks good to me” is none of them.

1. Automatic metrics (fast, noisy)

Eval loss — necessary but insufficient. Correlates weakly with downstream quality after ~1.5 perplexity.
ROUGE/BLEU — only valid for extractive tasks (summarization, translation). Useless for open-ended generation.
Format compliance rate — % of generations that parse as valid JSON / match regex / contain required tokens. Automatable, high signal for structured output tasks.

def format_compliance(generations, schema):
    """Returns fraction of generations matching JSON schema."""
    import json, jsonschema
    valid = 0
    for gen in generations:
        try:
            parsed = json.loads(extract_json(gen))
            jsonschema.validate(parsed, schema)
            valid += 1
        except Exception:
            pass
    return valid / len(generations)

2. LLM-as-judge (scalable, calibrated)

Use a strong model (GPT-4o, Claude 3.5 Sonnet, or a large open model) to score your fine-tuned outputs against a rubric. Pairwise comparison (fine-tuned vs. base instruct) is more reliable than absolute scoring.

JUDGE_PROMPT = """Compare two responses to the same prompt. 
Score 1 if Response A is better, -1 if Response B is better, 0 if tie.
Criteria: {criteria}

Prompt: {prompt}
Response A: {response_a}
Response B: {response_b}

Output only: 1, 0, or -1"""

Run 200–500 eval prompts. Bootstrap confidence intervals. If the fine-tuned model doesn’t beat base instruct with p < 0.05, don’t deploy it.

3. Human eval on the critical path (non-negotiable)

Sample 50–100 generations from your actual production distribution (not the eval set). Have domain experts rate: correctness, safety, tone, format. Track regression on a fixed “golden set” across every fine-tune run.

Pitfall: Evaluating only on the training distribution. Your eval set must include OOD prompts — typos, adversarial injections, multi-turn context, missing info — because that’s where fine-tuned models fail differently than base models.

Merge, quantize, serve

Merge LoRA weights for inference

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "./llama3-8b-lora/checkpoint-best")
model = model.merge_and_unload()  # fuses LoRA into base weights
model.save_pretrained("./llama3-8b-merged")
tokenizer.save_pretrained("./llama3-8b-merged")

Merged model = no PEFT dependency at inference, no adapter loading latency, standard generate() API.

Quantize for serving

For production throughput, quantize the merged model to AWQ 4-bit or GPTQ 4-bit (not the QLoRA NF4 used in training — different calibration).

# AWQ via autoawq — faster inference on GPU
python -m awq quantize \
  --model_path ./llama3-8b-merged \
  --quant_path ./llama3-8b-awq \
  --w_bit 4 --q_group_size 128 --zero_point --version GEMM

AWQ 4-bit typically retains 99%+ of BF16 quality at 2–3× throughput. Profile your specific workload — batch size, sequence length, KV cache pressure — before committing.

Serving stack

vLLM — best throughput for batched inference, PagedAttention, OpenAI-compatible API.
TGI (Text Generation Inference) — better for streaming, lower latency at batch=1, built-in watermarking.
llama.cpp / ollama — CPU/edge deployment, not for high-throughput GPU serving.

# vLLM OpenAI-compatible endpoint — drop-in for most clients
from vllm import LLM, SamplingParams

llm = LLM(
    model="./llama3-8b-awq",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
    max_model_len=8192,
    quantization="awq",
)

outputs = llm.generate(
    prompts=["<|begin_of_text|><|start_header_id|>user<|end_header_id|>Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>"],
    sampling_params=SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512),
)

Common failure modes and how to debug them

Symptom	Likely cause	Fix
Generates `assistant<\|end_header_id\|>` repeatedly	Chat template mismatch in training data	Re-verify `apply_chat_template` on every example
Loss drops but eval quality degrades	Overfitting to format, not task	Reduce epochs, increase dropout, add data diversity
Hallucinates new API endpoints	Knowledge injection attempt via SFT	Switch to RAG; fine-tune only for calling convention
Safe refusals on valid requests	Over-alignment from instruct base	DPO on preference pairs (accept vs. refuse)
KV cache OOM at batch > 4	`max_model_len` too high	Set `max_model_len=4096` or use sliding window
Throughput 10× lower than benchmark	Running BF16 merged model, not quantized	Deploy AWQ/GPTQ; verify `quantization` arg in vLLM

When to stop fine-tuning and try something else

Fine-tuning has diminishing returns. Consider alternatives when:

Data < 500 high-quality examples — few-shot prompting with a larger model (70B or 400B+) often wins.
Task changes weekly — RAG + prompt engineering adapts faster than retraining.
Need factual accuracy on evolving corpus — fine-tuning bakes in stale knowledge; retrieval doesn’t.
Latency budget < 100ms — even quantized 8B struggles; consider distillation to 1B–3B or a specialized encoder.
Multi-tenant with per-customer behavior — LoRA adapters per customer (swappable at inference) scale better than per-customer full fine-tunes.

Checklist before you call it done

Eval loss plateaued, train/eval gap < 0.1
LLM-as-judge shows statistically significant win over base instruct
Human eval on 50+ production-distribution samples: no regressions on safety, format, correctness
Format compliance rate > 99% on structured output tasks
Merged + AWQ model loads and generates in vLLM/TGI
Latency/throughput meets SLA at target batch size
Golden set regression test automated in CI
Rollback plan: base instruct model tagged and deployable in < 5 min

Fine-tuning Llama 3 is a lever, not a magic wand. The engineers who ship reliable models treat data as code, evaluation as testing, and deployment as a first-class concern — not an afterthought. Start small, measure ruthlessly, and only scale the recipe once the numbers justify it.