Decoder. plain-English AI glossary

Benchmark

● Core

A standard test that lets you compare how well different models solve the same problem.

Think of it like

Like a race track where every runner covers the same distance under the same conditions.

Example

MMLU is a benchmark of multiple-choice questions spanning history, math, science. Every model's score on MMLU is directly comparable.

How it actually works

Benchmarks democratize progress by making it measurable and reproducible. They're invaluable for tracking improvement, but they have downsides: models can overfit to them (especially if trained on leaked data), they might not correlate with real-world performance, and they can bake in existing biases. Good benchmarks are diverse, hard to game, and regularly retired as models solve them.

For product teams

Use benchmarks to track progress and communicate it clearly, but don't mistake benchmark scores for product quality.

For engineers

Evaluate on multiple benchmarks and always include held-out, human-annotated test sets.

Golden Dataset — Related evaluation artifact.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome