Decoder. plain-English AI glossary

Reward model eval

▲ Rising

Testing whether your preference model or reward model correctly ranks good outputs above bad ones.

Think of it like

Checking whether a referee's judgment matches actual game rules.

Example

You have pairs of responses (good, bad). Your reward model should score the good one higher. You measure what fraction it gets right.

How it actually works

Reward model evals check a different signal than generation evals. A reward model might rank A > B while a user would choose B > A. Reward models inherit biases from their training data — if trained on assistant-friendly feedback, they score conservative outputs high even when users want bold ones. This is why some teams use preference data directly instead of training an intermediate reward model.

For product teams

A biased reward model breaks your whole RLHF pipeline. Test it separately.

For engineers

Collect preference pairs. Measure accuracy, Kendall tau, or AUC. Stratify by example type.

Part of RLHF pipelines.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome