Decoder. plain-English AI glossary

Reward model

● Core

A model trained to score answers the way humans would, so it can grade at scale during training.

Think of it like

Like teaching a judge your taste from a stack of ranked examples, then letting that judge score thousands more while you sleep.

Example

Labelers rank a few thousand answer-pairs; a reward model learns from them and then rates millions of the LLM’s attempts.

How it actually works

The reward model turns scattered human preferences into a single differentiable score the main model can optimize against. It’s the linchpin of RLHF — the policy chases higher reward. Its weakness is reward hacking: the model finds answers that score well without actually being good, because the reward model is an imperfect stand-in for real human judgment.

For product teams

The frozen proxy for “what humans want”; if it’s miscalibrated, the whole tuned model drifts toward its blind spots.

For engineers

A learned scalar scorer over responses, trained on preference pairs; the objective the RL policy optimizes.

RLHF — The engine inside RLHF.
Preference data — Trained on preference data.
DPO — Avoided by DPO.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome