Can you combine RAG and fine-tuning together

Most teams treat retrieval-augmented generation and fine-tuning as mutually exclusive choices. They are not. The strongest production systems use fine-tuning to teach the model how to reason over retrieved context — formatting citations, suppressing hallucinations, handling contradictory passages — while RAG supplies the fresh, domain-specific knowledge the model never saw during training. This post walks through a concrete hybrid architecture you can deploy today.

Step 1: Define the division of labor

Before writing code, decide what each component owns. Fine-tuning should encode behavior: citation style, refusal tone, multi-hop reasoning patterns, and domain-specific terminology. RAG should own facts: product specs, regulatory text, customer histories, and anything that changes weekly. Write this contract down; it prevents scope creep when stakeholders ask for “just one more document type” in the fine-tuning data.

A practical split for a legal-assistant product:

Step 2: Curate a fine-tuning dataset that teaches retrieval hygiene

Your fine-tuning data must show the model how to consume retrieved chunks, not just how to answer from memory. Each training example needs three fields: the user question, the retrieved context (simulated), and the ideal response with inline citations.

{
  "messages": [
    {
      "role": "system",
      "content": "You are a legal research assistant. Cite every factual claim using [doc_id] format. If context is insufficient, say so explicitly."
    },
    {
      "role": "user",
      "content": "What is the holding in Smith v. Jones regarding implied warranty?"
    },
    {
      "role": "assistant",
      "content": "Smith v. Jones held that implied warranty extends to subsequent purchasers when the defect existed at time of sale [doc_3]. The court rejected privity as a requirement [doc_3]."
    }
  ]
}

Generate 2,000–5,000 such examples. Vary the retrieved context: sometimes complete, sometimes missing key paragraphs, sometimes contradictory. The model learns to say “The provided documents do not address X” instead of hallucinating. Use a script to simulate retrieval noise — drop 20% of relevant chunks, inject 10% irrelevant ones — so the fine-tuned model is robust to real RAG imperfections.

Step 3: Fine-tune a base model with LoRA

Full-parameter fine-tuning is rarely necessary. LoRA adapters (rank 16–32, alpha 32) on a 7B–70B base model converge in hours on a single A100 and preserve the model’s general reasoning. The following uses Hugging Face trl and peft; adjust model_id to your base (Llama-3-8B, Mistral-7B, Qwen2.5-14B).

# finetune_rag_adapter.py
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

dataset = load_dataset("json", data_files="rag_training.jsonl", split="train")

training_args = TrainingArguments(
    output_dir="./rag-lora-adapter",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=3,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    max_seq_length=4096,
    packing=True,
    dataset_text_field="messages",
)

trainer.train()
model.save_pretrained("./rag-lora-adapter-final")

Verify success: Run a 50-example held-out eval set. Check that citation format matches [doc_id] exactly, refusal rate on insufficient-context examples exceeds 90%, and no hallucinated case names appear. Log these metrics; they become your regression baseline.

Step 4: Build the retrieval pipeline with explicit contracts

Your retriever must return chunks in the exact format the fine-tuned model expects. If training used [doc_3] citations, the retriever must assign stable doc_id values and include them in the prompt. Use a hybrid search: dense embeddings for semantic match, BM25 for exact statute numbers, and a cross-encoder reranker for precision.

# retriever.py
from sentence_transformers import SentenceTransformer, CrossEncoder
from rank_bm25 import BM25Okapi
import numpy as np

class HybridRetriever:
    def __init__(self, corpus_path: str):
        self.docs = self._load_corpus(corpus_path)
        self.embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
        self.doc_embeddings = self.embedder.encode([d["text"] for d in self.docs], normalize_embeddings=True)
        self.bm25 = BM25Okapi([d["text"].split() for d in self.docs])

    def _load_corpus(self, path):
        # Returns list of {"doc_id": "doc_12", "text": "...", "metadata": {...}}
        ...

    def retrieve(self, query: str, k: int = 20, final_k: int = 6):
        # Dense retrieval
        q_emb = self.embedder.encode([query], normalize_embeddings=True)
        dense_scores = (self.doc_embeddings @ q_emb.T).squeeze()
        dense_idx = np.argsort(dense_scores)[-k:][::-1]

        # BM25 retrieval
        bm25_scores = self.bm25.get_scores(query.split())
        bm25_idx = np.argsort(bm25_scores)[-k:][::-1]

        # Merge candidates (reciprocal rank fusion)
        candidates = {}
        for rank, idx in enumerate(dense_idx):
            candidates[idx] = candidates.get(idx, 0) + 1.0 / (rank + 1 + 60)
        for rank, idx in enumerate(bm25_idx):
            candidates[idx] = candidates.get(idx, 0) + 1.0 / (rank + 1 + 60)

        fused_idx = sorted(candidates, key=candidates.get, reverse=True)[:50]

        # Cross-encoder rerank
        pairs = [[query, self.docs[i]["text"]] for i in fused_idx]
        rerank_scores = self.reranker.predict(pairs)
        reranked = sorted(zip(fused_idx, rerank_scores), key=lambda x: x[1], reverse=True)

        return [self.docs[i] for i, _ in reranked[:final_k]]

Verify success: Run 200 representative queries. Measure recall@6 against a labeled relevant-doc set. Target >0.85 recall@6. Log latency p50/p95; the reranker should add <150ms.

Step 5: Assemble the inference loop with citation enforcement

The inference loop stitches the retriever and fine-tuned model together. Critical detail: the prompt template must match the fine-tuning format exactly, including system prompt, doc_id injection, and any special tokens.

# inference.py
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

class RAGFineTunedPipeline:
    def __init__(self, base_model_id: str, adapter_path: str, retriever):
        self.tokenizer = AutoTokenizer.from_pretrained(base_model_id)
        self.tokenizer.pad_token = self.tokenizer.eos_token

        base = AutoModelForCausalLM.from_pretrained(
            base_model_id,
            torch_dtype=torch.bfloat16,
            device_map="auto",
            attn_implementation="flash_attention_2",
        )
        self.model = PeftModel.from_pretrained(base, adapter_path).eval()
        self.retriever = retriever

    def _build_prompt(self, question: str, docs: list) -> str:
        context_blocks = []
        for i, d in enumerate(docs):
            context_blocks.append(f"[doc_{i+1}] {d['text']}")
        context = "\n\n".join(context_blocks)

        messages = [
            {"role": "system", "content": "You are a legal research assistant. Cite every factual claim using [doc_id] format. If context is insufficient, say so explicitly."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ]
        return self.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

    @torch.inference_mode()
    def answer(self, question: str, max_new_tokens: int = 512) -> dict:
        docs = self.retriever.retrieve(question)
        prompt = self._build_prompt(question, docs)
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)

        outputs = self.model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            top_p=0.95,
            do_sample=True,
            pad_token_id=self.tokenizer.eos_token_id,
        )
        response = self.tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)

        # Verify citations reference only provided doc_ids
        cited = set()
        for tok in response.split():
            if tok.startswith("[doc_") and tok.endswith("]"):
                cited.add(tok[1:-1])
        provided = {f"doc_{i+1}" for i in range(len(docs))}
        invalid = cited - provided

        return {
            "answer": response,
            "citations": list(cited),
            "invalid_citations": list(invalid),
            "retrieved_docs": [d["doc_id"] for d in docs],
        }

Verify success: Run the 50-example eval set end-to-end. Assert invalid_citations == [] for every example. Measure citation precision (cited docs are actually relevant) and citation recall (all relevant docs cited). Target both >0.8.

Step 6: Add a fallback path for out-of-distribution queries

Fine-tuned models degrade gracefully on familiar patterns but can confidently hallucinate on truly novel questions. Add a lightweight classifier that routes low-confidence retrievals to a larger, un-fine-tuned model with a stricter system prompt. The classifier can be as simple as: if the top reranker score < 0.3 or retrieved chunk count < 3, escalate.

# fallback.py
def needs_fallback(retrieved_docs: list, rerank_scores: list) -> bool:
    if len(retrieved_docs) < 3:
        return True
    if max(rerank_scores) < 0.3:
        return True
    return False

The fallback model uses the same retriever output but a system prompt that forces “I cannot answer from the provided documents” when appropriate. This two-tier design keeps latency low for the 85% of queries in-distribution while protecting against the long tail.

Step 7: Instrument production observability

You cannot improve what you do not measure. Log every request with: query hash, retrieved doc_ids, rerank scores, generated answer, citation validity flag, latency breakdown (retrieve, rerank, generate), and user feedback (thumbs up/down). Build a daily dashboard showing:

• Citation validity rate (should stay >98%)
• Fallback trigger rate (should stay <15%; rising means retriever drift)
• Hallucination reports from user feedback
• P95 latency per component

# logging_middleware.py
import json
import time
import uuid
from contextlib import contextmanager

@contextmanager
def trace_request(query: str):
    request_id = str(uuid.uuid4())[:8]
    start = time.perf_counter()
    timings = {}
    try:
        yield timings
    finally:
        total = time.perf_counter() - start
        log_entry = {
            "request_id": request_id,
            "query_hash": hash(query),
            "total_latency_ms": round(total * 1000, 1),
            **{k: round(v * 1000, 1) for k, v in timings.items()},
        }
        print(json.dumps(log_entry))  # Ship to your log aggregator

Wrap each pipeline stage:

with trace_request(query) as t:
    t0 = time.perf_counter()
    docs, scores = retriever.retrieve_with_scores(query)
    t["retrieve"] = time.perf_counter() - t0

    if needs_fallback(docs, scores):
        t["fallback"] = True
        answer = fallback_model.answer(query, docs)
    else:
        t0 = time.perf_counter()
        answer = primary_pipeline.answer(query)
        t["generate"] = time.perf_counter() - t0

Step 8: Schedule continuous improvement loops

The hybrid system has two independent update cycles. RAG corpus updates daily — new cases, amended statutes, client documents. Fine-tuning updates quarterly — new citation formats, revised refusal policies, corrected failure modes from production logs.

Automate the fine-tuning retrain: collect all queries where invalid_citations != [] or user feedback flagged hallucination. Mine the retriever logs for queries with low rerank scores but high user satisfaction (the retriever missed something). Add these to the training set, regenerate simulated retrieval contexts, and re-run the LoRA training from the base model (not from the previous adapter — avoids catastrophic forgetting).

# retrain.sh - runs monthly via cron
python mine_failure_cases.py --logs s3://bucket/logs/ --output new_examples.jsonl
python augment_training_set.py --base rag_training.jsonl --new new_examples.jsonl --out rag_training_v2.jsonl
python finetune_rag_adapter.py --data rag_training_v2.jsonl --output ./rag-lora-adapter-v2
# Canary deploy: shadow 5% traffic, compare citation validity, promote if delta > -0.5%

Step 9: Evaluate the hybrid against pure baselines

Before declaring victory, run a controlled comparison on a held-out test set of 500 queries spanning in-distribution, near-OOD, and far-OOD. Compare three systems:

1. Pure RAG: Same retriever, base model with RAG prompt only
2. Pure fine-tune: Fine-tuned model, no retrieval (context window stuffed with full corpus where possible)
3. Hybrid: Your pipeline

Metrics: citation accuracy, hallucination rate (human-evaluated on 100 samples), refusal appropriateness, latency, and cost per query. The hybrid should dominate on hallucination rate and refusal appropriateness while matching pure RAG on latency. Pure fine-tune will fail on fresh facts; pure RAG will fail on citation discipline.

Step 10: Harden the deployment

Three operational details separate a demo from a production service:

1. Model version pinning: Deploy the fine-tuned adapter as an immutable artifact (Docker image or model registry version). Never hot-swap adapters in-place.
2. Retriever consistency: The retriever must return deterministic doc_id ordering for the same query. If you shard the index, include a tiebreaker (doc_id) in the sort key.
3. Cache-control: For repeated queries (common in legal workflows), cache the retrieved doc set and the generated answer separately. Invalidate on corpus update. If you route through a gateway like n4n.ai, forward the provider’s cache-control hints so upstream caches behave correctly.

# Dockerfile.adapter
FROM python:3.11-slim
COPY rag-lora-adapter-final/ /model/
COPY inference.py /app/
RUN pip install --no-cache-dir torch transformers peft accelerate flash-attn
ENTRYPOINT ["python", "/app/inference.py"]

Build, tag with git SHA, push to registry. Your orchestration pulls by tag; rollback is kubectl set image deployment/rag-api rag-api=registry/rag-adapter:v1.2.3-abc1234.

The hybrid approach works because it respects what each technique is good at. Fine-tuning compresses behavior into weights; RAG expands knowledge at query time. The engineering work is in the interfaces — the citation schema, the retriever contract, the fallback trigger — not in chasing marginal benchmark gains on either component alone. Ship the loop, measure the right things, and iterate.