Win Rate
In pairwise comparisons, the percentage of times judges prefer one model over another—intuitive and human-aligned.
Think of it like
Like head-to-head match results in a tournament: Model A won 70% of matches against Model B.
Example
LLM judges compare 100 outputs from Model A and Model B. Model A wins 65 comparisons. Win rate for Model A is 65%.
How it actually works
Win rate is more interpretable than absolute scores—'70% better' is clearer than 'score of 7.3 vs 6.1.' It's also more robust to score calibration issues. Downside: pairwise comparison is expensive and requires a good judge. Transitive comparisons can give surprising results (Model A > B, B > C, but C > A is possible).
For product teams
Win rate is the most interpretable metric for comparing models to stakeholders.
For engineers
Implement pairwise evaluation; use multiple judges to reduce noise.
Related
- Elo Rating — Metric it uses.
- Pairwise Comparison — How it's computed.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome