Decoder. plain-English AI glossary

LLM-as-Judge

▲ Rising

Using one language model to evaluate the quality of another model's output.

Think of it like

Like asking a published author to review another author's manuscript.

Example

Instead of paying humans to grade summaries, you prompt GPT-4 to rate whether another model's summary is accurate and concise.

How it actually works

LLM-as-judge is fast and scales, but it's not objective. The judge model has its own biases and errors. It can be gamed if you know what it values. It works best for subjective tasks (is this helpful?) where multiple reasonable answers exist. For factual tasks (is this true?), you need grounding in external sources.

For product teams

Use LLM judges to rapidly iterate on model quality, but validate against human raters regularly.

For engineers

Implement judge consistency checks; compare LLM scores to human scores on a holdout set.

Human Eval — The opposite approach.
Win Rate — Metric it often computes.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome