Pairwise Comparison
Comparing two models side-by-side on the same input and asking judges which output is better.
Think of it like
Like a taste test where you try both sodas and pick your favorite.
Example
You show judges the same prompt and outputs from Model A and Model B. They pick which response is clearer, more helpful, more accurate.
How it actually works
Pairwise comparison is more reliable than absolute scoring because judges anchor on relative differences, not absolute scales. Psychologically, humans are better at comparisons than independent ratings. Downside: it's expensive (n models = n² comparisons) and you can't measure degradation on single metrics (only relative ranking).
For product teams
Pairwise comparison is the most reliable way to measure relative model quality.
For engineers
Collect multiple judgments per pair to reduce noise; record model order (A vs B vs B vs A) to detect biases.
Related
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome