Factuality
Whether the model's statements are true according to objective reality or provided context.
Think of it like
Like checking if a news story is true by verifying it against other sources.
Example
Model says 'Paris is the capital of France.' Factuality: true. Model says 'Paris is the capital of Germany.' Factuality: false.
How it actually works
Factuality is hard to measure for open-domain queries because truth is often subjective or requires world knowledge. For closed-domain (summarizing documents, fact-checking), you can measure it automatically. Models trained on web data can hallucinate common false beliefs. RAG and fact-checking improve factuality.
For product teams
Factuality is non-negotiable for products that inform decisions. Never ship factually unreliable models.
For engineers
Measure factuality on gold sets; use RAG or retrieval-augmented generation; add fact-checking guardrails.
Related
- Hallucination Rate — Specific instance of lack of this.
- Faithfulness — Related metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome