Elo Rating
A rating system that ranks models based on pairwise wins and losses, accounting for opponent strength.
Think of it like
Like chess ratings: beating a stronger opponent gains more points than beating a weaker one.
Example
Model A has a 1600 Elo rating. If it beats Model B (rated 1500) in a comparison, it gains rating points.
How it actually works
Elo is more sophisticated than raw win percentage because it adjusts for who you beat. It's used in competition leaderboards (like LMSys Arena) to rank models dynamically. Updates dynamically as new comparisons come in. The math is probabilistic—your rating predicts your win probability against other models.
For product teams
Elo ratings are intuitive for stakeholders and handle dynamic model releases well.
For engineers
Use established Elo implementations; be transparent about update frequency and match selection bias.
Related
- Pairwise Comparison — What it measures.
- Arena — Elo leaderboards use.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome