Eval-driven development
Writing evals first, before you build the feature, then iterating to pass them.
Think of it like
Test-driven development for language models — write the test, then write the code.
Example
You need a system to summarize documents. You write 50 examples with gold summaries and grade them on ROUGE score. Then you iterate on prompts until your ROUGE improves.
How it actually works
Eval-driven development forces clarity about what "done" means. It makes progress visible and prevents shipping half-baked features. The danger: optimizing the metric instead of the real problem. A high ROUGE score doesn't guarantee users like the summary.
For product teams
Speeds up iteration and keeps teams aligned. Requires discipline to not game the metrics.
For engineers
Write evals in a CI pipeline. Gate merges on eval improvements. Track score history.
Related
- Philosophical approach to quality.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome