Annotator agreement
Also called Inter-Annotator Agreement
How often different human labelers give the same answer — a health check on whether your labels mean anything.
Think of it like
Two referees watching the same replay: if they keep disagreeing on the call, the rulebook is the problem.
Example
When only 60% of annotators agree on which reply is "better," the team rewrites the guidelines before trusting the data.
How it actually works
If humans labeling the same examples do not agree, the task is ambiguous or the instructions are unclear, and a model trained on that noisy data inherits the confusion. Agreement is measured with scores like Cohen's or Fleiss' kappa that correct for chance. Low agreement is a signal to clarify guidelines or accept that the task has irreducible subjectivity — high agreement is a prerequisite for trustworthy labels.
For product teams
A quick gauge of whether your labeled data is signal or noise before you build on it.
For engineers
Chance-corrected concordance (e.g. kappa) among labelers; low values indicate ambiguous tasks or guidelines.
Related
- Annotation — The activity it measures the quality of.
- Labeling — A related labeling process.
- Preference data — A dataset whose quality it guards.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome