Evals
Also called Evaluations
Tests for AI — a repeatable way to check whether the model actually got better or just feels like it did.
Think of it like
A unit-test suite, but for judgment instead of code.
Example
Before shipping a new prompt, a team runs it against 500 saved questions and counts how many it nails.
How it actually works
They range from exact-match graders to “LLM-as-judge” setups where one model scores another. The hard part is never running them — it is writing eval sets that reflect what real users actually do, not what is easy to grade.
For product teams
A regression suite for quality — proof a change actually moved the metric before you ship.
For engineers
Automated graders (exact-match, rubric, or LLM-as-judge) scoring output over a fixed dataset.
Related
- HITL — The scoreboard a human in the loop quietly fills.
- Hallucination — What you run to catch a hallucination before users do.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome