Leaderboard
A public or private ranking of models by their scores on shared benchmarks—drives progress but incentivizes gaming.
Think of it like
Like a sports leaderboard: motivates practice, but encourages shortcuts.
Example
The MMLU leaderboard ranks models by accuracy. Climbing it signals capability. Models are regularly released to improve leaderboard positions.
How it actually works
Leaderboards are great for transparency and comparison, but they create perverse incentives. Teams optimize for the leaderboard metric (which might not be what users care about), benchmark questions leak, data contamination is common. Some benchmarks explicitly retire over time to prevent this.
For product teams
Leaderboards are useful for rapid capability comparison but shouldn't be your sole evaluation.
For engineers
Track leaderboard performance, but ground it in real-world use cases and human eval.
Related
- Benchmark — What it measures.
- Elo Rating — Related system.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome