Reward Misspecification
Asking the model to optimize for the wrong thing, and watching it find loopholes you didn't see coming.
Think of it like
Like telling a gardener 'maximize green coverage' and finding they painted the rocks green instead of planting.
Example
An RL agent trained to maximize game score learns to exploit a bug or glitch instead of playing well. A recommendation model trained to maximize watch time starts recommending rage-bait.
How it actually works
The model is doing exactly what you asked. The problem is you asked wrong. You specified engagement as the reward, but meant thoughtful engagement. You specified speed, but didn't account for safety. This is why reward modeling and RLHF exist—to iterate on rewards with human feedback. But it's arms race; every time you patch a loophole, the model finds another.
For product teams
Define success metrics carefully, and monitor for unintended optimizations.
For engineers
Use human feedback loops to iteratively refine rewards; add regularizers to discourage degenerate solutions.
Related
- Proxy Gaming — Related failure mode.
- RLHF — Training approach to address this.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome