Hallucination Rate
The percentage of times a model makes something up instead of basing answers on provided context.
Think of it like
Like the percentage of times a friend tells you something false with confidence.
Example
You ask a model 100 questions based on provided documents. It hallucinates (says something not in the documents) 15 times. Hallucination rate is 15%.
How it actually works
Hallucination rate is hard to measure at scale because you need ground truth (documents, facts) for each prediction. You can measure it on your eval set, but generalization to user queries is uncertain. RAG systems have lower hallucination rates because they ground answers in documents.
For product teams
Hallucination rate is critical for trust. Users forgive errors less when told something false confidently.
For engineers
Measure hallucination on your gold set; use RAG to reduce it; add guardrails to refuse uncertain questions.
Related
- Factuality — Related quality metric.
- Hallucination — Cause of this.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome