Decoder. plain-English AI glossary

HumanEval

● Core

A benchmark of 164 Python coding problems where correctness is unambiguous—the standard for measuring code generation.

Think of it like

Like a programming interview where either your code runs and passes tests, or it doesn't.

Example

Problem: 'Write a function that returns the nth Fibonacci number.' The model generates code; you run it against test cases.

How it actually works

HumanEval is small but hard. Most frontier models score 80–95%. It's valuable because code correctness is verifiable and unambiguous. Downside: it's only Python, only small problems, and overfitting is a risk (the dataset is public and finite). Pass@k is often reported alongside exact-match to show how many attempts the model needs.

For product teams

HumanEval is the quickest way to compare code generation capability. High HumanEval correlates with ability to write real code.

For engineers

Report both pass@1 and pass@10; compare to human baseline (humans get ~80% at pass@1).

Benchmark — What it measures.
Pass at k — Metric often used with it.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome