RLHF
Also called Reinforcement Learning from Human Feedback
Teaching a model to behave the way people prefer by turning human ratings into a training signal.
Think of it like
Like training a dog with treats — you can’t script every trick, so you reward the behaviors you want and it learns the pattern.
Example
Humans rank several model answers best-to-worst; those rankings train a reward model, which then guides the language model toward preferred responses.
How it actually works
RLHF collects human preference comparisons, fits a reward model to predict them, then uses reinforcement learning to nudge the base model toward high-reward outputs. It’s what turns a raw next-token predictor into a helpful, harmless assistant — but it can be gamed (reward hacking) and inherits the biases of whoever did the rating.
For product teams
The step that makes a model feel aligned and usable; also where your values and rater choices get encoded.
For engineers
Fit a reward model to human preference data, then optimize the policy against it (often PPO) with a KL penalty to the base.
Related
- Pretraining — What it refines.
- Instruction following — The behavior it instills.
- Learning signal — The preference signal it turns into training.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome