Answer Relevance
Whether the model's answer actually addresses the question or request.
Think of it like
Like ordering pizza and getting a good meal, but not pizza.
Example
Question: 'What's the capital of France?' Answer: 'The Eiffel Tower is famous.' Answer relevance: low (didn't answer the question).
How it actually works
Answer relevance is distinct from factuality (your answer can be true but irrelevant). It's easy to spot in evals but hard to define algorithmically. LLM judges can evaluate relevance, but they're not perfect. Robust question-answering systems check both relevance and factuality.
For product teams
Answer relevance is table stakes. Never ship a model that ignores questions.
For engineers
Measure on your gold set; use LLM judges or human raters; add safeguards to refuse off-topic questions.
Related
- Context Relevance — Complementary metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome