How RLHF turns a base model into ChatGPT

RLHF explained simply: it’s the three-stage pipeline that converts a raw next-token predictor into a model that follows instructions, refuses harmful requests, and maintains coherent multi-turn conversations. Base models know language; RLHF aligns them to human intent. This guide walks through each stage with the decisions that actually matter in production.

Stage 1: supervised fine-tuning creates the instruction-following backbone

Before any reinforcement learning, you need a supervised fine-tuning (SFT) phase. The base model has never seen a conversation format — it only knows document completion. SFT teaches the chat template, instruction following, and basic refusal behavior.

Data composition matters more than volume

You don’t need millions of examples. The original InstructGPT paper used ~13k prompts with human-written demonstrations. What you do need: diversity across task types (coding, creative writing, analysis, refusal), consistent formatting, and high-quality human annotations. Synthetic data from stronger models helps bootstrap, but human-written demonstrations remain the gold standard for the seed set.

# Typical SFT formatting for chat templates
def format_chatml(messages: list[dict]) -> str:
    """Convert OpenAI-style messages to ChatML format."""
    formatted = []
    for msg in messages:
        role = msg["role"]
        content = msg["content"]
        formatted.append(f"<|im_start|>{role}\n{content}<|im_end|>")
    formatted.append("<|im_start|>assistant\n")  # generation prompt
    return "\n".join(formatted)

# Example training sample
sample = {
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a Python function to parse CSV."},
        {"role": "assistant", "content": "import csv\n\ndef parse_csv(path):\n    with open(path) as f:\n        reader = csv.DictReader(f)\n        return list(reader)"}
    ]
}

Common pitfall: overfitting to the SFT format

If you train too long on SFT, the model becomes brittle — it follows the template perfectly but loses the broad knowledge from pretraining. Monitor validation loss on a held-out pretrain distribution (e.g., a slice of C4 or RedPajama) alongside your SFT validation set. Stop when SFT validation plateaus but pretrain validation hasn’t degraded significantly.

Rule of thumb: 1-3 epochs on high-quality SFT data. Larger models (70B+) need fewer epochs than 7B models.

Stage 2: reward modeling learns human preferences

The reward model (RM) is a scalar function that scores (prompt, completion) pairs. It replaces human judgment in the RL loop. Getting this right is the highest-leverage step — a misaligned reward model produces a misaligned policy no matter how well PPO optimizes it.

Preference data collection: the annotation protocol

Collect pairwise comparisons: annotators see a prompt and two model completions (side by side, randomized order), then choose which is better. Ties are allowed. Critical design decisions:

• Source of completions: Sample from multiple checkpoints (base, SFT, earlier RL checkpoints) and temperatures to cover the quality spectrum
• Annotator guidelines: Explicit rubrics for “helpful,” “harmless,” “honest” — vague instructions produce noisy labels
• Calibration: Run regular agreement checks (Cohen’s kappa) and re-annotate low-agreement batches

# Preference dataset structure
preference_sample = {
    "prompt": "Explain quantum entanglement to a 10-year-old.",
    "chosen": "Imagine two magic dice that always show the same number...",
    "rejected": "Quantum entanglement is a physical phenomenon where...",
    "metadata": {
        "annotator_id": "ann_042",
        "task_category": "explanation",
        "difficulty": "easy"
    }
}

Reward model architecture and training

The standard approach: initialize from the SFT model (or a slightly smaller variant), replace the LM head with a scalar head, and train with pairwise cross-entropy loss (Bradley-Terry model).

import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM

class RewardModel(nn.Module):
    def __init__(self, model_name: str, dropout: float = 0.1):
        super().__init__()
        self.backbone = AutoModelForCausalLM.from_pretrained(model_name)
        # Freeze early layers optionally
        hidden_size = self.backbone.config.hidden_size
        self.score_head = nn.Sequential(
            nn.Dropout(dropout),
            nn.Linear(hidden_size, 1)
        )
    
    def forward(self, input_ids, attention_mask=None):
        outputs = self.backbone(
            input_ids=input_ids,
            attention_mask=attention_mask,
            output_hidden_states=True,
            return_dict=True
        )
        # Use last hidden state of the last token
        last_hidden = outputs.hidden_states[-1][:, -1, :]
        return self.score_head(last_hidden).squeeze(-1)

# Pairwise loss (Bradley-Terry)
def reward_loss(chosen_rewards: torch.Tensor, rejected_rewards: torch.Tensor) -> torch.Tensor:
    # -log(sigmoid(chosen - rejected))
    return -torch.log(torch.sigmoid(chosen_rewards - rejected_rewards)).mean()

Critical: reward model overfitting and reward hacking

The RM will overfit to the preference dataset if trained too long. It learns spurious correlations — verbosity, specific phrases, formatting quirks — that correlate with high ratings in your annotation pool but don’t generalize.

Mitigations:

• Hold out a validation set from a different annotator pool
• Track reward variance across prompts — collapse indicates overfitting
• Use ensemble of 3-5 RMs and average scores (reduces variance ~√n)
• Regularize with KL penalty against the SFT model during RM training

# Ensemble inference
def ensemble_reward(models: list[RewardModel], input_ids, attention_mask=None):
    scores = []
    for model in models:
        with torch.no_grad():
            scores.append(model(input_ids, attention_mask))
    return torch.stack(scores).mean(dim=0)

Stage 3: PPO fine-tunes the policy against the reward model

Proximal Policy Optimization (PPO) updates the SFT model to maximize RM score while staying close to the original policy via KL penalty. This is where most engineering effort concentrates — PPO is notoriously unstable at scale.

The PPO loop in practice

from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer

# Setup
model = AutoModelForCausalLMWithValueHead.from_pretrained("sft-checkpoint")
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained("sft-checkpoint")  # frozen
tokenizer = AutoTokenizer.from_pretrained("base-model")
tokenizer.pad_token = tokenizer.eos_token

config = PPOConfig(
    batch_size=256,
    mini_batch_size=64,
    learning_rate=1.41e-5,
    kl_penalty="kl",        # "kl" or "kl_penalty"
    init_kl_coef=0.2,       # tune this aggressively
    target_kl=6.0,          # target KL divergence per update
    cliprange=0.2,
    cliprange_value=0.2,
    vf_coef=0.1,
    horizon=10000,
    gamma=1.0,
    lam=0.95,
)

ppo_trainer = PPOTrainer(config, model, ref_model, tokenizer)

# Generation + reward scoring + PPO step
for epoch in range(num_epochs):
    for batch in prompt_dataloader:
        # 1. Generate responses
        query_tensors = batch["input_ids"]
        response_tensors = ppo_trainer.generate(
            query_tensors,
            max_new_tokens=256,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
        )
        
        # 2. Score with reward model
        texts = [tokenizer.decode(q + r) for q, r in zip(query_tensors, response_tensors)]
        rewards = reward_model.score(texts)  # returns list of scalars
        
        # 3. PPO step
        stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
        
        # 4. Log everything
        log_stats(stats, rewards, epoch)

KL penalty: the single most important hyperparameter

The KL coefficient controls how far the policy drifts from the SFT model. Too low → reward hacking, gibberish, collapse. Too high → no improvement over SFT.

Practical tuning protocol:

1. Start with init_kl_coef=0.2 and target_kl=6.0
2. Monitor three metrics per step: mean_reward, mean_kl, entropy
3. If mean_kl > target_kl consistently → increase init_kl_coef
4. If mean_reward plateaus but mean_kl << target_kl → decrease init_kl_coef
5. Target KL of 2-10 is typical for 7B-70B models; larger models tolerate lower KL

# Adaptive KL control (common in production)
def update_kl_coef(trainer, current_kl, target_kl=6.0, lr=0.01):
    if current_kl > target_kl * 1.5:
        trainer.kl_coef *= 1.5
    elif current_kl < target_kl * 0.5:
        trainer.kl_coef *= 0.5
    trainer.kl_coef = max(0.01, min(trainer.kl_coef, 1.0))

Common PPO failure modes

Reward clipping and normalization

Raw RM scores vary wildly across prompts. Normalize per-batch:

def normalize_rewards(rewards: torch.Tensor, eps: float = 1e-8) -> torch.Tensor:
    """Standardize rewards within batch."""
    return (rewards - rewards.mean()) / (rewards.std() + eps)

# Clip extreme values to prevent gradient spikes
rewards = torch.clamp(normalize_rewards(raw_rewards), -4, 4)

Evaluation: how to know it actually works

RLHF explained in papers often skips the evaluation rigor required in production. You need three evaluation axes:

1. Automated benchmarks (necessary but insufficient)

Run MMLU, GSM8K, HumanEval, BBH, and your domain-specific evals. Compare against the SFT checkpoint — not the base model. RLHF should maintain or improve knowledge benchmarks while improving instruction following.

2. Reward model evaluation (circular but useful)

Evaluate the final policy on a held-out preference set using the same RM. Win rate vs. SFT should be >60%. But: this measures RM alignment, not human alignment.

3. Human evaluation (the only thing that matters)

Side-by-side A/B tests: your RLHF model vs. SFT vs. a strong baseline (GPT-4, Claude). Minimum 200 prompts covering your target use cases. Measure:

• Preference win rate
• Refusal rate on harmful prompts (should be high)
• False refusal rate on benign prompts (should be low)
• Multi-turn coherence (run 5-turn conversations)

# Side-by-side evaluation harness
async def evaluate_pairwise(
    model_a, model_b, prompts: list[str], judge_model, n_trials=3
):
    """Run model_a vs model_b with model-based judge."""
    results = {"a_wins": 0, "b_wins": 0, "ties": 0}
    
    for prompt in prompts:
        for _ in range(n_trials):
            # Randomize order
            a_first = random.random() > 0.5
            resp_a = await model_a.generate(prompt)
            resp_b = await model_b.generate(prompt)
            
            judgment = await judge_model.judge(
                prompt, resp_a, resp_b, a_first=a_first
            )
            
            if judgment == "A":
                results["a_wins" if a_first else "b_wins"] += 1
            elif judgment == "B":
                results["b_wins" if a_first else "a_wins"] += 1
            else:
                results["ties"] += 1
    
    return results

Tradeoffs you’ll face in production

Data quality vs. quantity

You can scale preference data with synthetic generation (using a stronger model to label), but the annotation protocol for your seed set determines the ceiling. Invest in 5k-10k high-quality human preferences before scaling synthetically.

Model size vs. RLHF difficulty

Larger models (70B+) are harder to RLHF, not easier. They have sharper reward landscapes, more severe reward hacking, and require more careful KL control. 7B-13B models are more forgiving for iteration.

Online vs. offline RLHF

Standard PPO is online — you generate from the current policy. Offline methods (DPO, IPO, KTO) optimize directly on a static preference dataset without generation in the loop. They’re more stable and cheaper but can’t explore beyond the dataset.

# DPO loss (offline alternative) - simpler, no RM needed
def dpo_loss(
    policy_chosen_logps: torch.Tensor,
    policy_rejected_logps: torch.Tensor,
    ref_chosen_logps: torch.Tensor,
    ref_rejected_logps: torch.Tensor,
    beta: float = 0.1
) -> torch.Tensor:
    """Direct Preference Optimization loss."""
    policy_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps - ref_rejected_logps
    logits = policy_logratios - ref_logratios
    losses = -torch.log(torch.sigmoid(beta * logits))
    return losses.mean()

When to use DPO: First alignment pass, limited compute, stable preferences. When to use PPO: You need to explore beyond your preference data, have compute for online generation, or need iterative improvement loops.

Deployment considerations

The RLHF model serves the same API as the base model but with different behavior. If you’re running an inference gateway that routes across 240+ models, the RLHF checkpoint becomes just another model variant — same endpoint, same token accounting, same fallback logic. The routing layer doesn’t care how the model was trained, only that it honors the chat template and returns valid completions.

Checklist before you ship

• SFT validation loss plateaued without pretrain degradation
• Reward model validation accuracy >75% on held-out annotators
• RM ensemble variance <0.1 on calibration prompts
• PPO KL divergence stable in target range (2-10) for 3+ checkpoints
• No reward hacking patterns in generated samples (manual spot-check 100+)
• Human eval win rate >55% vs. SFT on target use cases
• False refusal rate <5% on benign edge cases
• Multi-turn coherence passes 5-turn conversation test
• Inference latency within SLA (RLHF adds no latency vs. SFT)

RLHF is not magic — it’s a disciplined pipeline where each stage constrains the next. The reward model encodes your values; PPO optimizes for them; evaluation catches the gaps. Skip the rigor at any stage and you’ll ship a model that looks good on benchmarks but fails users in production.