What is DPO? Direct preference optimization explained

Direct Preference Optimization (DPO) is a method for aligning language models to human preferences without reinforcement learning. Instead of training a reward model and then optimizing a policy against it with PPO, DPO directly optimizes the policy on a dataset of ranked completions using a simple classification loss. The result is a simpler, more stable training pipeline that matches or exceeds RLHF quality with less compute and fewer moving parts.

How DPO works

RLHF traditionally has three stages: supervised fine-tuning (SFT), reward model training, and policy optimization via PPO. DPO collapses the last two stages into one. The key insight: the optimal policy under a Bradley-Terry preference model has a closed-form solution that can be optimized directly with a contrastive loss on preference pairs.

Given a prompt x and two completions y_w (preferred) and y_l (dispreferred), the DPO loss is:

L_DPO(π_θ; π_ref) = -E_{(x, y_w, y_l) ~ D} [ log σ( β * (log π_θ(y_w|x) - log π_ref(y_w|x) - log π_θ(y_l|x) + log π_ref(y_l|x)) ) ]

Where:

• π_θ is the policy being trained
• π_ref is the reference model (typically the SFT model)
• β is a temperature parameter controlling deviation from the reference
• σ is the sigmoid function

This loss pushes the policy to assign higher probability to preferred completions and lower probability to dispreferred ones, relative to the reference model. The reference model acts as an implicit reward model — its log-probabilities serve as the baseline.

The derivation in brief

Under the Bradley-Terry model, the probability that y_w is preferred over y_l given x is:

P(y_w > y_l | x) = σ( r(x, y_w) - r(x, y_l) )

Where r is the reward function. The optimal policy for a given reward under a KL constraint has the form:

π*(y|x) = (1/Z) π_ref(y|x) exp( (1/β) r(x, y) )

Rearranging gives r(x, y) = β log(π*(y|x) / π_ref(y|x)) + β log Z. Substituting into the preference probability and dropping the constant yields the DPO loss. No reward model training required.

Why DPO matters for practitioners

Simpler infrastructure

PPO requires maintaining four models during training: policy, reference, reward model, and value network. It needs rollouts, advantage estimation, and careful hyperparameter tuning (clipping, KL penalties, learning rates). DPO needs two models (policy and reference) and a static preference dataset. You can train it with standard supervised learning tooling — no RL libraries, no rollout workers, no reward model serving.

More stable training

PPO suffers from reward hacking, training collapse, and sensitivity to hyperparameters. The policy can drift into regions where the reward model gives high scores but outputs are nonsensical. DPO’s implicit reward is anchored to the reference model’s log-probabilities, providing a natural regularizer. The KL constraint is baked into the loss formulation rather than enforced via a penalty term that must be tuned.

Better data efficiency

DPO learns directly from preference comparisons. Each training example provides a gradient signal for both the preferred and dispreferred completion. PPO’s reward model learns a scalar score per completion, then the policy learns from sampled rollouts — two stages of approximation. Empirically, DPO reaches comparable alignment quality with fewer preference labels.

Easier debugging

When PPO produces bad outputs, the failure could be in the reward model, the value network, the advantage estimator, or the policy update. With DPO, you inspect the loss on your preference pairs. If the model assigns higher probability to dispreferred completions, the loss tells you directly. You can also evaluate the implicit reward r(x, y) = β log(π_θ(y|x) / π_ref(y|x)) on held-out data to verify alignment.

Concrete training example

Suppose you’re aligning a 7B parameter model for code generation. You have an SFT checkpoint and a dataset of 10,000 preference pairs collected from developers choosing between two model completions for the same prompt.

Data format

Each example in your preference dataset:

{
  "prompt": "Write a Python function that retries an async HTTP request with exponential backoff.",
  "chosen": "async def fetch_with_retry(url, max_retries=3, base_delay=1.0):\n    for attempt in range(max_retries):\n        try:\n            return await http_get(url)\n        except TransientError as e:\n            if attempt == max_retries - 1:\n                raise\n            await asyncio.sleep(base_delay * (2 ** attempt))\n    raise RuntimeError('unreachable')",
  "rejected": "def fetch_with_retry(url):\n    while True:\n        try:\n            return requests.get(url)\n        except:\n            time.sleep(1)"
}

Training loop (PyTorch)

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("my-sft-checkpoint")
ref_model = AutoModelForCausalLM.from_pretrained("my-sft-checkpoint")
ref_model.eval()
for p in ref_model.parameters():
    p.requires_grad = False

tokenizer = AutoTokenizer.from_pretrained("my-sft-checkpoint")
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-7)
beta = 0.1

def dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta):
    policy_logratios = policy_chosen_logps - policy_rejected_logps
    ref_logratios = ref_chosen_logps - ref_rejected_logps
    logits = beta * (policy_logratios - ref_logratios)
    return -F.logsigmoid(logits).mean()

def get_logprobs(model, input_ids, attention_mask, labels):
    outputs = model(input_ids=input_ids, attention_mask=attention_mask)
    logits = outputs.logits[:, :-1, :]
    labels = labels[:, 1:]
    logprobs = F.log_softmax(logits, dim=-1)
    token_logprobs = torch.gather(logprobs, 2, labels.unsqueeze(-1)).squeeze(-1)
    mask = (labels != -100).float()
    return (token_logprobs * mask).sum(dim=1) / mask.sum(dim=1)

for batch in dataloader:
    # Tokenize chosen and rejected separately
    chosen_enc = tokenizer(batch["chosen"], padding=True, truncation=True, max_length=2048, return_tensors="pt")
    rejected_enc = tokenizer(batch["rejected"], padding=True, truncation=True, max_length=2048, return_tensors="pt")
    
    # Labels: -100 for prompt tokens, token ids for completion tokens
    # (assumes you've constructed labels that mask the prompt)
    
    policy_chosen_logps = get_logprobs(model, **chosen_enc)
    policy_rejected_logps = get_logprobs(model, **rejected_enc)
    
    with torch.no_grad():
        ref_chosen_logps = get_logprobs(ref_model, **chosen_enc)
        ref_rejected_logps = get_logprobs(ref_model, **rejected_enc)
    
    loss = dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rejected_logps, beta)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Key implementation details

Reference model frozen: The reference model stays at the SFT checkpoint throughout training. This is critical — if you update it, you lose the anchor that prevents reward hacking.

Beta tuning: β controls how far the policy can deviate from the reference. Typical values: 0.1–0.5. Lower β = stronger KL constraint = more conservative updates. Higher β = more aggressive optimization but risk of overfitting to preferences. Start at 0.1 and monitor the implicit reward gap on a validation set.

Sequence-level logprobs: The loss operates on average token log-probability per sequence. This is different from token-level losses in SFT. Make sure your logprob computation masks prompt tokens correctly.

Length normalization: The average logprob formulation naturally handles variable-length completions. However, if your chosen/rejected pairs have systematically different lengths, consider adding a length penalty or filtering pairs with extreme length ratios.

Mixed precision: DPO trains well with bfloat16 on H100s/A100s. Gradient accumulation works normally — the loss is a scalar per batch.

Common misconceptions

“DPO is just supervised learning on chosen completions”

False. If you only train on chosen completions (SFT on the preferred responses), you ignore the signal in the rejected completions. DPO’s contrastive loss uses both: it increases probability of chosen relative to rejected and relative to the reference model. Training only on chosen data produces a model that mimics the preferred style but doesn’t learn to avoid the specific failure modes present in rejected completions.

“You need a reward model for evaluation”

You don’t. The implicit reward r(x, y) = β log(π_θ(y|x) / π_ref(y|x)) is a valid reward function. You can evaluate alignment by computing this reward on held-out prompts and comparing chosen vs. rejected completions. A well-aligned model should assign higher implicit reward to chosen completions. This also lets you detect reward hacking: if implicit reward increases but human evaluation doesn’t, your preference data may have systematic biases.

“DPO requires on-policy data”

DPO is an off-policy algorithm. The preference dataset can be collected from any policy — human annotators comparing outputs from GPT-4, Claude, your SFT model, or a mix. The reference model in the loss is your SFT checkpoint, not the policy that generated the data. This is a major practical advantage: you can reuse existing preference datasets without regenerating them as your policy improves.

“Beta is just a learning rate”

β is not a learning rate. It’s the inverse temperature of the KL constraint in the constrained optimization problem that DPO derives from. The learning rate controls step size in parameter space; β controls the shape of the objective. You still need a learning rate (typically 1e-6 to 5e-7 for 7B models). Changing β changes what optimum you converge to, not how fast you get there.

“DPO replaces SFT”

DPO starts from an SFT checkpoint. The reference model is the SFT model. If you skip SFT and run DPO from a base model, the reference model provides no useful anchor — base models assign near-uniform probability to most completions, so the implicit reward signal is noise. SFT teaches the model the output format and basic competence; DPO teaches it which of two competent completions is better.

“DPO can’t handle ties or multi-way preferences”

The standard DPO loss assumes binary preferences. For ties, you can either filter them out or use a modified loss that treats ties as equal probability (logsigmoid(0) = -log 2). For k-way preferences (ranking 3+ completions), you can decompose into pairwise comparisons or use a listwise loss like Listwise DPO. The pairwise decomposition works well in practice and keeps the implementation simple.

When to use DPO vs. PPO

Iterative DPO

In production, you’ll likely run multiple DPO rounds. After each round, the trained policy becomes the new reference model for the next round, and you collect fresh preferences using the updated policy. This is the DPO analog of iterative RLHF.

# Round 1
ref_model = sft_checkpoint
policy = train_dpo(ref_model, preferences_v1)

# Round 2
ref_model = policy  # previous round's policy
policy = train_dpo(ref_model, preferences_v2)  # new preferences from updated model

Each round typically uses a lower β (e.g., 0.1 → 0.05 → 0.01) since the policy is already better aligned and needs less constraint. Monitor the implicit reward on a held-out validation set — if it plateaus or reverses, stop.

Summary

DPO replaces the PPO stage of RLHF with a direct classification loss on preference pairs. It eliminates the reward model, value network, and rollout infrastructure while matching or exceeding alignment quality. For most teams aligning open-source models, DPO is the default choice — simpler to implement, easier to debug, and more stable to train. The reference model anchor prevents reward hacking without manual KL tuning. Start with β=0.1, use your SFT checkpoint as the reference, and iterate with fresh preference data.