Reward model training
Also called Reward Model Training
Teaching a separate model to predict which responses humans prefer, so it can score outputs during RL.
Think of it like
Training a judge by showing them thousands of pairs of dishes and which one diners liked, until they can score new plates.
Example
From a dataset of chosen-vs-rejected reply pairs, a model learns to output higher scores for the responses people picked.
How it actually works
The reward model is trained on human preference comparisons to output a scalar "how good is this" score. In RLHF, that score becomes the reward signal a policy optimizes against. Its quality caps the whole pipeline: if it is exploitable, the policy will reward-hack it, and if its preferences are biased or narrow, those flaws propagate into the final model. It usually starts from the base model itself.
For product teams
The stand-in for human judgment that scales preference feedback — and whose blind spots become the model's.
For engineers
Train a scalar scorer on pairwise human preferences; it provides the reward in RLHF and is a target for reward hacking.
Related
- Reward model — The model it produces.
- Preference data — The data it learns from.
- RLHF — The pipeline it feeds.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome