Capability evaluation
Testing whether a model can perform a specific skill (math, code, reasoning) at all, regardless of perfect execution.
Think of it like
A driver's test where you just check whether someone can parallel park, not how perfectly.
Example
Does the model know how to do chain-of-thought reasoning on math problems? You ask a few hard math questions and see if it attempts reasoning steps.
How it actually works
Capability eval answers "can this model do the thing?" Benchmark eval answers "how well?" Capabilities emerge unpredictably as models scale, so capability eval is how researchers discover new abilities. The catch: a model might have the capability but fail due to prompting, temperature settings, or just bad luck on the sample.
For product teams
Tells you what to expect from a model before detailed benchmarking.
For engineers
Write tests for each capability. Use simple examples first. Track emergence curves as models scale.
Related
- Precursor to detailed evaluation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome