Decoder. plain-English AI glossary

Preference optimization

▲ Rising

Also called Preference Optimization

Training a model directly from "this answer is better than that one" comparisons to make preferred outputs more likely.

Think of it like

Teaching taste by always pointing at the better of two plates, until the cook internalizes what you like.

Example

Instead of running full RL, a team applies DPO on a dataset of chosen-vs-rejected pairs and the model shifts toward the chosen style.

How it actually works

This family reframes alignment as learning from pairwise preferences. Classic RLHF does it indirectly — train a reward model, then optimize with RL. Newer methods like DPO skip the separate reward model and the RL loop, deriving an objective that directly raises the probability of preferred responses over dispreferred ones. Simpler and more stable, though sometimes less flexible than full RL.

For product teams

The trend that made preference-based training cheaper and more reliable to run.

For engineers

Optimize a policy on chosen/rejected pairs; DPO-style objectives bypass an explicit reward model and RL loop.

DPO — The method that popularized skipping RL.
Preference data — The data it consumes.
RLHF — The RL-based approach it competes with.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome