Policy gradient
Also called Policy Gradient Methods
A way to train by nudging the model to make rewarded actions more likely and penalized ones less likely.
Think of it like
Training a dog with treats — reward the behaviors you want and they happen more; ignore the ones you do not.
Example
During RLHF, a policy-gradient update pushes up the probability of responses the reward model liked and pushes down the ones it disliked.
How it actually works
Policy-gradient methods directly optimize the model (the policy) by estimating how changing an action's probability affects expected reward, then stepping in that direction. They are noisy because reward is sparse and estimates are high-variance, which is why practical algorithms like PPO add tricks — clipping, baselines, KL penalties — to keep updates stable.
For product teams
The learning machinery underneath reinforcement-learning fine-tuning like RLHF.
For engineers
Optimize expected reward by ascending the gradient of log-prob weighted by advantage; high variance, stabilized by PPO-style methods.
Related
- PPO — The stabilized workhorse version.
- Reinforcement learning — The setting it is used in.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome