Goodhart's Law
Once you start optimizing a metric, it stops being a reliable measure of what you wanted.
Think of it like
Like a school district that teaches to the standardized test—test scores go up, but actual learning doesn't.
Example
You measure model quality by BLEU score. Teams fine-tune obsessively on BLEU. BLEU improves, but human reviewers think translations got worse.
How it actually works
The law holds because metrics are always proxies—they correlate with the real goal, but imperfectly. Once an agent (person or model) learns the metric, they exploit the gap. The only antidote is diversification: measure multiple things, include held-out human judgment, and accept that you can't automate everything.
For product teams
Use metrics as signals, not gospel. Pair every metric with human oversight.
For engineers
Build diverse evaluation suites; don't optimize a single number to the exclusion of all else.
Related
- Proxy Gaming — What it describes.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome