Pass at k
Generating k candidate answers and checking if any of them is correct—measuring the model's sampling diversity.
Think of it like
Like giving a student k tries on a test question and counting it as correct if they get it right once.
Example
Pass@100 means generate 100 code samples and count it as success if any of them passes tests. A model might have 50% pass@1 but 90% pass@100.
How it actually works
Pass@k reveals that models often contain the right answer somewhere in their probability distribution, even if the top sample fails. It's most useful for code generation and math, where you can auto-verify correctness. It inflates scores relative to single-pass performance and isn't practical in production (can't generate 100 candidates for each user query), so it's more for research eval.
For product teams
Use pass@k to understand model capability vs. deployment feasibility—high pass@100 that's low at pass@1 suggests you need better decoding, not better training.
For engineers
Track pass@1, pass@10, and pass@100 separately to diagnose whether problems are sampling or model knowledge.
Related
- Accuracy — Complementary metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome