DPO tuned preferences without a separate reward model

Skipping the reward model and the RL loop

The RLHF pipeline described in the InstructGPT paper fits a separate reward model to human preference comparisons, then uses reinforcement learning, specifically proximal policy optimisation, to adjust the language model against that reward model. The Direct Preference Optimization paper, submitted in May 2023, reparameterises the same preference-modelling objective so that the optimal policy can be derived in closed form, replacing the reward-model-plus-reinforcement-learning pipeline with a single classification loss trained directly on preference pairs, with no sampling from the model required during training.

What the paper claimed relative to RLHF

The paper reports that its method matched or exceeded RLHF's summarisation and single-turn dialogue quality on the benchmarks it tested, and reports better control over generated sentiment than a PPO-based RLHF baseline, while describing the training procedure as simpler to implement and requiring substantially less hyperparameter tuning than the two-stage pipeline it replaces. These are comparisons against the specific RLHF implementation and datasets the paper used; a different reward model, a different preference dataset, or a different RL implementation is a separate comparison the paper does not make.

What practitioners later found

The method's own reference implementation has since added variants such as conservative DPO, which adds a label-smoothing term to account for preference labels that may be flipped or noisy, and a related identity-preference-optimisation variant, changes made after the original release rather than described in the original paper. Their existence in the maintained repository indicates practitioners found the original loss sensitive to noisy or low-quality preference data in ways the initial paper did not fully address, though the repository itself does not quantify how often that sensitivity matters in practice. Whether DPO or an RLHF-style pipeline performs better on a specific model and preference dataset is a comparison that has to be run on that pairing; the paper's reported results are specific to its own base models and data.

• Were the preference pairs used for training checked for label noise, and does the training method account for it?
• Is the comparison against RLHF using a reward model and RL implementation of comparable quality, or a weaker baseline?
• Does the reported quality hold on multi-turn tasks, or only on the single-turn settings the original paper tested?

The closed-form derivation is a genuine simplification of the RLHF objective; whether it matches RLHF's output quality on a given model and dataset remains an empirical question the paper answers only for its own experiments.