Grader
A function or model that takes a model's output and a reference and produces a score—automating the evaluation step.
Think of it like
Like an automated essay grader that reads your essay and assigns a score.
Example
Grader function: takes model output and gold answer, returns 1 if correct, 0 if wrong. Used in eval pipeline.
How it actually works
Graders can be rule-based (exact match, length checks) or learned (LLM-as-judge, a fine-tuned model trained to score outputs). Learned graders are more flexible but require training data. Graders should be fast (not calling APIs repeatedly) and deterministic when possible.
For product teams
Good graders are force-multipliers. They let you evaluate at scale.
For engineers
Build deterministic rule-based graders for simple metrics; train graders for complex judgments.
Related
- Automated Grading — Used in.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome