Ensemble evaluation
Using multiple evaluators (models or humans) to grade the same output and averaging their scores.
Think of it like
Having three teachers grade the same essay and averaging the scores.
Example
Running Claude, GPT-4, and Llama against your outputs and averaging their ratings of relevance from 1–10.
How it actually works
Ensemble eval reduces bias from any single grader. But it's expensive and sometimes the models agree because they're trained on similar data. For critical applications, combining automated scoring with human spot-checks is better than either alone.
For product teams
More reliable than single-judge evals but also more expensive. Use for milestones, not daily testing.
For engineers
Call multiple eval models, collect scores, return mean and std. Log disagreements for audit trails.
Related
- Hardens eval reliability.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome