How ChatGPT was trained with RLHF

Understanding how ChatGPT uses RLHF requires looking past the marketing summaries and into the three-stage pipeline that actually produces the model: supervised fine-tuning on demonstration data, training a reward model on comparison data, and policy optimization via PPO. Each stage introduces distinct failure modes that still show up in production behavior today. This post walks through the mechanics, the data flows, and the practical constraints that shaped the final system.

The three-stage pipeline

The RLHF process for ChatGPT (and InstructGPT before it) follows a sequence that has become standard across the industry: SFT → RM → PPO. Each stage consumes the output of the previous one, and each stage has different data requirements, compute profiles, and failure modes.

Base model (GPT-3.5)
       │
       ▼
┌──────────────────┐
│  Stage 1: SFT    │  ← Demonstration data (prompt → ideal response)
│  (supervised     │
│   fine-tuning)   │
└──────────────────┘
       │
       ▼
┌──────────────────┐
│  Stage 2: RM     │  ← Comparison data (prompt → response A vs B → label)
│  (reward model)  │
└──────────────────┘
       │
       ▼
┌──────────────────┐
│  Stage 3: PPO    │  ← Prompts only (policy generates, RM scores, update)
│  (RL optimization)│
└──────────────────┘
       │
       ▼
   ChatGPT

This isn’t theoretical — it’s the exact dependency chain. You cannot train the reward model without the SFT model generating candidates, and you cannot run PPO without a frozen reward model providing scalar feedback.

Stage 1: Supervised fine-tuning on demonstrations

The SFT stage takes the base language model and fine-tunes it on a dataset of (prompt, ideal_response) pairs. For ChatGPT, OpenAI collected roughly 13,000–15,000 demonstrations from labelers who were instructed to write high-quality responses following specific guidelines.

The training objective is standard causal language modeling:

def sft_loss(model, batch):
    # batch: {"input_ids": [...], "labels": [...]}
    # labels = input_ids shifted right, with -100 for prompt tokens
    logits = model(batch["input_ids"]).logits
    shift_logits = logits[:, :-1, :].contiguous()
    shift_labels = batch["labels"][:, 1:].contiguous()
    loss = F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_labels.view(-1),
        ignore_index=-100
    )
    return loss

Key implementation details that matter:

Prompt masking: Only response tokens contribute to the loss. The prompt tokens are masked with -100 (PyTorch’s ignore_index). This prevents the model from “learning” the prompt distribution, which it already knows from pretraining.

Data quality over quantity: 13k examples is tiny compared to pretraining (trillions of tokens). The leverage comes from distribution shift — these examples target the instruction-following capability specifically. Labelers were given detailed guidelines: “be helpful and harmless,” “don’t make up facts,” “ask clarifying questions when ambiguous.”

No RL yet: At this stage, the model learns to mimic good responses. It does not learn to prefer them over bad ones. That distinction matters — SFT models often hallucinate confidently because they’ve never seen negative examples.

Stage 2: Training the reward model

The reward model (RM) is a scalar function r(prompt, response) → ℝ that predicts human preference. Training it requires comparison data: for a given prompt, labelers rank multiple model outputs.

Data collection

For each prompt, the SFT model generates K responses (typically K=4–9). Labelers rank them from best to worst. This produces (K choose 2) pairwise comparisons per prompt. With ~33k prompts and K=4, that’s roughly 100k–200k comparisons.

# Simplified comparison dataset structure
comparison = {
    "prompt": "Explain quantum entanglement simply",
    "response_a": "Quantum entanglement is when...",  # ranked 1st
    "response_b": "It's like magic particles...",      # ranked 3rd
    "response_c": "Entanglement means...",            # ranked 2nd
    "response_d": "I don't know much about this.",    # ranked 4th
    "ranking": [0, 2, 1, 3]  # indices sorted by quality
}

Model architecture

The RM is typically the SFT model with the language modeling head replaced by a single linear layer projecting to a scalar. The final hidden state of the last token (or a pooled representation) feeds this head.

class RewardModel(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.base = base_model
        # Freeze most layers initially, or use LoRA
        self.reward_head = nn.Linear(base_model.config.hidden_size, 1)
    
    def forward(self, input_ids, attention_mask):
        outputs = self.base(input_ids=input_ids, attention_mask=attention_mask)
        last_hidden = outputs.last_hidden_state  # [batch, seq, hidden]
        # Use last token's hidden state (assumes right padding)
        last_token_idx = attention_mask.sum(dim=1) - 1
        pooled = last_hidden[torch.arange(last_hidden.size(0)), last_token_idx]
        reward = self.reward_head(pooled).squeeze(-1)
        return reward

Training objective: Bradley-Terry

The standard loss models the probability that response i beats response j as a logistic function of their reward difference:

def reward_loss(model, batch):
    # batch contains multiple responses per prompt
    # shape: [batch_size, num_responses, seq_len]
    rewards = model(batch["input_ids"], batch["attention_mask"])
    # rewards: [batch_size, num_responses]
    
    # For each pair (i, j) where i is preferred over j:
    # log σ(r_i - r_j)
    loss = 0
    n_pairs = 0
    for b in range(rewards.size(0)):
        ranking = batch["ranking"][b]  # e.g., [0, 2, 1, 3]
        for idx_i, idx_j in itertools.combinations(range(len(ranking)), 2):
            # ranking[idx_i] is preferred over ranking[idx_j]
            pref_idx = ranking[idx_i]
            dispref_idx = ranking[idx_j]
            diff = rewards[b, pref_idx] - rewards[b, dispref_idx]
            loss += -F.logsigmoid(diff)
            n_pairs += 1
    return loss / n_pairs

Practical issues

Length bias: RMs consistently prefer longer responses. The fix is length normalization — either during training (add length penalty to loss) or at inference (divide reward by token count^α).

Calibration drift: The RM’s absolute scores mean nothing; only relative ordering matters. But PPO treats them as absolute. This mismatch causes reward hacking (see Stage 3).

Labeler disagreement: On subjective tasks (creative writing, tone), labelers disagree ~30–40% of the time. The Bradley-Terry model assumes a latent “true” ranking. In practice, you either accept noise or model annotator uncertainty explicitly (e.g., with a Thurstone model).

Stage 3: PPO optimization

Proximal Policy Optimization updates the SFT policy to maximize the RM score while staying close to the original SFT model (via KL penalty). This is where “how ChatGPT uses RLHF” becomes an engineering balancing act.

The PPO objective

For each prompt, the current policy generates a response. The RM scores it. The policy is updated to increase expected reward:

def ppo_step(policy, ref_policy, reward_model, batch_prompts, 
             clip_eps=0.2, kl_coef=0.04, lr=1e-6):
    # Generate responses from current policy
    responses = policy.generate(batch_prompts, max_new_tokens=512)
    
    # Score with reward model
    rewards = reward_model(responses)
    
    # KL penalty against reference (SFT) policy
    with torch.no_grad():
        ref_logprobs = ref_policy.get_logprobs(batch_prompts, responses)
    curr_logprobs = policy.get_logprobs(batch_prompts, responses)
    kl = (curr_logprobs - ref_logprobs).sum(dim=-1).mean()
    
    # PPO clipped objective
    # Advantage = reward - baseline (often just reward - mean_reward)
    advantages = rewards - rewards.mean()
    ratio = torch.exp(curr_logprobs.sum(-1) - old_logprobs.sum(-1))
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
    policy_loss = -torch.min(surr1, surr2).mean()
    
    # Total loss
    loss = policy_loss + kl_coef * kl
    loss.backward()
    optimizer.step()

Critical hyperparameters

Parameter	Typical range	Effect
`kl_coef`	0.01–0.1	Higher = stays closer to SFT, less reward gain
`clip_eps`	0.1–0.3	Prevents catastrophic policy shifts
`lr`	1e-6 – 1e-5	Must be tiny; policy collapses fast otherwise
`rollout_batch_size`	64–512	Larger = better gradient estimates

The KL constraint is the whole game

Without the KL penalty, the policy quickly discovers adversarial inputs that maximize the RM but produce garbage — “reward hacking.” The RM is an imperfect proxy; optimizing it too hard breaks the actual behavior you want.

# What reward hacking looks like in practice
# RM score goes up, human eval goes down
# Example: model learns to append "I hope this helps!" to every response
# because labelers slightly preferred polite endings

OpenAI’s InstructGPT paper reports using kl_coef ≈ 0.04 with a target KL of ~0.6 nats per token. In practice, you monitor KL per token during training and adjust kl_coef dynamically:

def adjust_kl_coef(current_kl, target_kl=0.6, lr=0.01):
    # Simple PID-style adjustment
    error = current_kl - target_kl
    return kl_coef * (1 + lr * error)

Where the pipeline breaks

Each stage has characteristic failure modes that propagate forward.

SFT failures → RM failures

If SFT demonstrations contain hallucinations (they do), the RM learns to reward confident-sounding hallucinations because labelers often can’t verify factual claims in real time. The RM then reinforces this during PPO.

RM failures → PPO failures

The RM is a classifier, not a generator. It learns “what looks like a good answer” not “what is a good answer.” Classic examples:

Prefers longer responses (length bias)
Prefers hedging language (“It’s important to note that…”)
Rewards sycophancy — agreeing with the user’s premise even when wrong

PPO instability

PPO on LLMs is notoriously unstable. Common symptoms:

Reward collapse: KL explodes, reward spikes, generations become incoherent
Mode collapse: Policy converges to a few safe templates (“I don’t have enough information…”)
Training divergence: Loss goes NaN from logprob underflow

Mitigations that actually work:

Gradient clipping at 1.0 (non-negotiable)
Logprob clipping before exponentiation: torch.clamp(logprob, -20, 0)
Early stopping on validation KL, not training reward
Multiple RM checkpoints — ensemble scores reduce variance

DPO: The simpler alternative

Direct Preference Optimization (DPO) eliminates the RM and PPO stages entirely. It optimizes the policy directly on preference data using a closed-form solution derived from the Bradley-Terry model.

def dpo_loss(policy, ref_policy, batch, beta=0.1):
    # batch: prompt, chosen_response, rejected_response
    policy_chosen_logps = policy.get_logprobs(batch["prompt"], batch["chosen"])
    policy_rejected_logps = policy.get_logprobs(batch["prompt"], batch["rejected"])
    ref_chosen_logps = ref_policy.get_logprobs(batch["prompt"], batch["chosen"])
    ref_rejected_logps = ref_policy.get_logprobs(batch["prompt"], batch["rejected"])
    
    # Log ratios
    policy_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps - ref_rejected_logps
    
    # DPO loss: -log σ(β * (policy_logratios - ref_logratios))
    logits = beta * (policy_logratios - ref_logratios)
    loss = -F.logsigmoid(logits).mean()
    return loss

DPO advantages:

No RM training (saves ~30% compute)
No PPO instability (no online RL loop)
Simpler hyperparameter space (mainly β)

DPO disadvantages:

Requires the reference policy to be the SFT model (can’t iterate)
Less flexible — can’t incorporate multiple reward signals easily
Theoretical gap: DPO assumes the RM is optimal under Bradley-Terry; PPO doesn’t

For teams building their first aligned model, DPO is usually the right starting point. PPO makes sense when you have mature RM infrastructure and need to combine multiple objectives (helpfulness + safety + style).

Data flywheels and production reality

The published pipeline describes a static training run. Production systems like ChatGPT add continuous feedback loops:

User feedback (thumbs up/down) → new comparison data
Automated filters (refusal detection, toxicity) → synthetic preference pairs
Periodic retraining — weekly or monthly cycles incorporating new data

# Simplified production data pipeline
def collect_preference_data(logs, sample_rate=0.01):
    pairs = []
    for log in logs.sample(frac=sample_rate):
        if log.user_rating is not None:
            # Explicit feedback
            pairs.append(PreferencePair(
                prompt=log.prompt,
                chosen=log.response if log.user_rating > 0 else None,
                rejected=log.response if log.user_rating < 0 else None,
                source="explicit"
            ))
        elif log.auto_flags:
            # Implicit: model refused but user re-asked → preferred alternative
            pairs.append(PreferencePair(
                prompt=log.prompt,
                chosen=log.next_response,
                rejected=log.response,
                source="implicit"
            ))
    return pairs

This flywheel is where the real alignment gains happen. The initial RLHF run gets you to “mostly helpful, mostly harmless.” The flywheel handles edge cases, new capabilities, and distribution shift.

Honest assessment

If you’re building an RLHF pipeline today, here’s the practical priority order:

Get SFT data right — 10k high-quality demonstrations beat 100k noisy ones. Invest in labeler guidelines and calibration.
Build the RM carefully — Length normalization, calibrate on held-out human judgments, ensemble multiple checkpoints.
Start with DPO — Skip PPO unless you have a specific reason (multi-objective, existing RM infrastructure).
Instrument everything — Track KL/divergence, reward distributions, generation quality metrics per prompt category.
Plan the flywheel from day one — Logging infrastructure for implicit/explicit feedback is not optional.

The core insight: RLHF is not a training algorithm. It’s a data pipeline with an RL step at the end. The model quality is bounded by your demonstration and comparison data quality. No amount of PPO tuning compensates for labelers who don’t understand the task.

Takeaway: How ChatGPT uses RLHF is a three-stage pipeline where the data collection and reward modeling decisions matter far more than the PPO implementation details. DPO has largely superseded PPO for new projects — same data, fewer moving parts, fewer ways to fail. If you’re running PPO in 2024, you should have a written justification for why DPO doesn’t meet your requirements.