PPO
Also called Proximal Policy Optimization
The reinforcement-learning algorithm that powered classic RLHF, designed to improve the model without letting each update lurch too far.
Think of it like
Adjusting a recipe one careful pinch at a time — never dumping in a whole cup and ruining the batch.
Example
A lab fine-tunes a chat model with PPO against a reward model, keeping a KL leash so it stays close to the original model.
How it actually works
PPO is a policy-gradient method that caps how much the policy can change per step, using a clipped objective and a KL penalty against a reference model. That restraint is what makes RL on language models stable enough to work. It is effective but fiddly — many moving parts and hyperparameters — which is part of why simpler alternatives like DPO gained ground.
For product teams
The engine behind the first wave of aligned chat models; powerful but operationally heavy.
For engineers
Clipped-objective policy gradient with a KL penalty to a reference policy, stabilizing RLHF updates.
Related
- Policy gradient — The family of methods it belongs to.
- RLHF — The training regime it implements.
- KL divergence — The leash that keeps it close to the base model.
- DPO — The simpler alternative that skips the RL loop.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome