Reinforcement learning from human feedback (RLHF) is a technique that aligns language models to human preferences by training a reward model on comparative human judgments, then optimizing the policy against that reward model using reinforcement learning. It replaces the brittle heuristics of supervised fine-tuning with a learned objective that captures nuanced qualities like helpfulness, honesty, and safety. The result is a model that follows instructions more reliably and refuses inappropriate requests without explicit rule lists.
How RLHF works in three phases
RLHF decomposes into three distinct training stages. Each stage has different data requirements, compute profiles, and failure modes.
Phase 1: Supervised fine-tuning (SFT)
Start with a pre-trained base model. Collect a dataset of high-quality prompt–response pairs written by human annotators or distilled from stronger models. Fine-tune the base model on this dataset using standard cross-entropy loss.
# Typical SFT loss — nothing RL-specific here
def sft_loss(logits, labels, mask):
shift_logits = logits[:, :-1, :].contiguous()
shift_labels = labels[:, 1:].contiguous()
shift_mask = mask[:, 1:].contiguous()
loss_fct = nn.CrossEntropyLoss(reduction='none')
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
loss = (loss * shift_mask.view(-1)).sum() / shift_mask.sum()
return lossSFT teaches the model the format of instruction following: how to structure answers, adopt a helpful tone, and recognize task boundaries. It does not teach the model to choose between competing valid responses — that comes next.
Data quality matters more than quantity here. A few thousand carefully curated examples often outperform hundreds of thousands of noisy ones. Teams typically use 10k–100k examples for SFT.
Phase 2: Reward model training
Collect comparative data: for a given prompt, present annotators with two or more model completions and ask them to rank or score them. Train a scalar reward model $r_\phi(x, y)$ to predict which completion humans prefer.
# Bradley-Terry style pairwise loss for reward modeling
def reward_model_loss(chosen_rewards, rejected_rewards, margin=0.0):
# chosen_rewards: [batch], rejected_rewards: [batch]
diff = chosen_rewards - rejected_rewards
loss = -F.logsigmoid(diff - margin).mean()
return lossThe reward model is typically a smaller transformer (1B–7B parameters) initialized from the SFT model with a regression head. It learns to assign higher scores to responses humans prefer. Key practical details:
- Comparison format: Side-by-side ranking scales better than absolute scoring. Annotators are more consistent at “A is better than B” than “A is 7/10.”
- Calibration: Raw reward model scores drift during RL training. Most pipelines normalize rewards per-prompt or use a KL penalty to prevent exploitation.
- Multi-objective rewards: Production systems often train separate reward heads for helpfulness, harmlessness, and style, then combine them with learned or fixed weights.
Phase 3: Policy optimization (PPO)
Optimize the SFT model (now the policy) to maximize the reward model’s output while staying close to the original SFT distribution. Proximal Policy Optimization (PPO) is the standard algorithm.
# Simplified PPO step for language models
def ppo_step(policy, ref_policy, reward_model, prompts, responses, old_logprobs):
# Current policy logprobs
new_logprobs = policy.get_logprobs(prompts, responses)
# Reference policy logprobs (frozen SFT model)
with torch.no_grad():
ref_logprobs = ref_policy.get_logprobs(prompts, responses)
# Reward from reward model
with torch.no_grad():
rewards = reward_model(prompts, responses)
# KL penalty
kl = (old_logprobs - ref_logprobs).sum(dim=-1)
# PPO clipped objective
ratio = torch.exp(new_logprobs - old_logprobs)
surr1 = ratio * rewards
surr2 = torch.clamp(ratio, 1 - eps, 1 + eps) * rewards
policy_loss = -torch.min(surr1, surr2).mean()
kl_loss = kl_coeff * kl.mean()
loss = policy_loss + kl_loss
loss.backward()
return loss.item()The KL penalty (or constraint) is critical. Without it, the policy exploits the reward model — generating gibberish that scores high but is useless to humans. Typical KL coefficients range from 0.01 to 0.1 depending on how much drift you tolerate.
PPO for LLMs differs from standard RL in two ways:
- Episode = one completion. No multi-step horizons, no discounting.
- Action space = vocabulary. The policy emits tokens autoregressively; PPO operates on the sequence-level reward.
Training takes 1–3 epochs over the prompt dataset (typically 50k–200k prompts). You monitor reward model score, KL divergence, and human evaluation metrics. Stop when reward plateaus or KL exceeds threshold.
Why RLHF matters for deployed systems
Base models predict plausible continuations. SFT models mimic instruction-following format. RLHF models optimize for human preference. The distinction shows up in production:
Instruction adherence: An SFT model answers “Write a poem without the letter e” with a poem that usually contains e. An RLHF model learns that constraint satisfaction is heavily rewarded and actually avoids the letter.
Refusal behavior: Base models complete harmful prompts. SFT models sometimes refuse, sometimes comply — depending on whether the training data covered that pattern. RLHF models learn a consistent refusal boundary because annotators consistently rank refusals higher for unsafe prompts.
Style control: You can steer verbosity, formality, or reasoning depth by adjusting reward model weights or adding preference data for the target style. No re-training of the base model required.
Calibration: RLHF reduces hallucination on factual questions if the reward model penalizes confident wrong answers. This requires annotators who can verify claims — expensive but effective.
Concrete example: From base to RLHF
Consider the prompt: Explain quantum entanglement to a 10-year-old.
Base model completion (truncated):
Quantum entanglement is a physical phenomenon that occurs when pairs or groups of particles are generated, interact, or share spatial proximity in a way such that the quantum state of each particle cannot be described independently of the state of the others, even when the particles are separated by a large distance…
Technically correct. Completely inappropriate for the audience.
SFT model completion:
Imagine you have two magic coins. When you flip them, they always land the same way — both heads or both tails — even if you take one to the moon! That’s kind of like quantum entanglement. The particles stay connected no matter how far apart they are.
Better format. But it might add “Scientists call this spooky action at a distance” — a cliché the training data reinforced.
RLHF model completion:
Picture two dice that always match. Roll one in New York, the other in Tokyo — same number, every time. Quantum particles can do this. They become linked so measuring one instantly tells you the other’s state, no matter the distance. Einstein called it “spooky action at a distance.” We use this for quantum computers and ultra-secure communication.
The RLHF version: leads with a concrete analogy, includes the historical hook, mentions applications, stops before the math. Annotators consistently ranked this style higher — accessible but not dumbed down, accurate but not dense. The reward model internalized that preference distribution.
Common misconceptions
“RLHF is just supervised learning on better data”
False. SFT minimizes cross-entropy against a fixed target distribution. RLHF optimizes a learned reward function that captures comparative preferences. The reward model generalizes to prompts not in the preference data. SFT cannot do this — it only interpolates training examples.
Evidence: InstructGPT (1.3B parameters with RLHF) outperformed GPT-3 (175B parameters, SFT only) on human evaluations despite 100x fewer parameters. The reward model provided a denser, more generalizable signal than imitation.
“The reward model is the objective”
The reward model is a proxy for human preferences. It has blind spots, biases, and exploitation vulnerabilities. Treating it as ground truth causes reward hacking — the policy finds inputs that maximize the reward model score while violating the actual intent.
Mitigations:
- KL regularization keeps the policy near the SFT distribution where the reward model is reliable
- Reward model ensembles detect disagreement as uncertainty
- Periodic human evaluation catches drift early
- Adversarial data collection — generate edge cases, get human labels, retrain reward model
“RLHF requires massive human annotation budgets”
The original InstructGPT used ~40k SFT examples, ~33k comparison rankings, and ~31k PPO prompts. Modern open-source pipelines (Zephyr, Tulu, UltraFeedback) achieve strong results with 10k–50k total human labels by:
- Using stronger models (GPT-4, Claude) to generate initial completions and even synthetic preferences
- Active learning — only querying humans on high-disagreement or high-uncertainty comparisons
- Multi-task reward models that share representations across objectives
A team of 5–10 annotators with clear guidelines can produce sufficient data for a 7B–13B model in weeks, not months.
“DPO replaces RLHF”
Direct Preference Optimization (DPO) reparameterizes the RLHF objective as a supervised loss on preference pairs, eliminating the reward model and PPO loop. It’s simpler, more stable, and often matches PPO performance — but it’s still RLHF in the sense of learning from human feedback via preference optimization. The distinction is algorithmic, not philosophical.
DPO trades off:
- Pros: No reward model training, no PPO hyperparameters, no KL tuning, faster iteration
- Cons: Requires the reference policy to be the SFT model (can’t start from arbitrary checkpoints), less flexible for multi-objective rewards, harder to incorporate offline RL techniques
For most teams starting today, DPO is the better default. PPO remains useful when you need fine-grained control over the reward composition or want to integrate with existing RL infrastructure.
Practical considerations for engineering teams
Infrastructure: RLHF needs three model copies in memory during PPO (policy, reference, reward model) plus a value head. For a 7B model, that’s ~60GB VRAM with 8-bit quantization — doable on 8×A100. DPO only needs policy + reference (~30GB).
Evaluation: Automated benchmarks (MT-Bench, AlpacaEval, MMLU) correlate with human preference but diverge on style and safety. Budget for weekly human eval on a held-out prompt set (200–500 prompts, 3–5 annotators each). Track win rate vs. SFT baseline and vs. previous RLHF checkpoint.
Data flywheel: Log production completions (with user consent). Flag low-reward or high-disagreement samples for annotation. Retrain reward model monthly. Retrain policy quarterly. This compounds — each cycle improves the reward model, which improves the policy, which generates better data.
Serving: The RLHF model is your production model. No separate alignment layer needed. However, you may want a lightweight classifier on top for real-time safety filtering (e.g., detecting prompt injection) since the reward model’s safety head isn’t calibrated for threshold-based blocking.
When to skip RLHF
- Narrow domain, fixed tasks: If you only need SQL generation or summarization with strict format requirements, SFT + few-shot prompting often suffices. RLHF’s generality isn’t worth the complexity.
- No annotation capacity: If you cannot access consistent human labelers (internal or vendor), synthetic preference data from GPT-4-class models works surprisingly well — but you’re distilling their alignment, not creating your own.
- Rapid prototyping: DPO or even best-of-n sampling against a reward model gives 80% of the benefit in 20% of the time. Start there.
RLHF is the reason deployed LLMs feel like assistants instead of autocomplete engines. It converts human judgment into a differentiable objective that shapes every token the model emits. The technique is mature enough that the limiting factor for most teams isn’t algorithmic — it’s annotation quality, evaluation rigor, and the discipline to maintain the data flywheel.