Self-consistency check
Running the same prompt multiple times and checking whether the model gives the same answer each time.
Think of it like
Asking a friend the same question three times and seeing if they answer consistently.
Example
You ask a model "Is 17 prime?" ten times with temperature 1.0. If it says "yes" eight times and "no" twice, that's concerning — the true answer should be consistent.
How it actually works
Self-consistency uses the model to grade itself. High variance across runs suggests either ambiguous inputs or hallucination. The catch: multiple outputs that agree can all be wrong if the model is confidently biased. You're measuring consistency, not correctness.
For product teams
A cheap signal that something might be wrong, but not a sign that something is right.
For engineers
Run k times, check whether outputs match. Use embedding similarity for fuzzy matching on non-binary answers.
Related
- Related pattern for confidence estimation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome