Online evaluation
Measuring how your system performs on real user traffic after it's deployed.
Think of it like
A test drive with real traffic, not an empty parking lot.
Example
After deploying a new ranking model, you measure how often users click on the top result or use the search again immediately.
How it actually works
Online eval sees real behavior but it's slow and messy. Users don't always click the right result, and clicks can mean "interesting" instead of "relevant." You need statistical rigor to tell signal from noise. And online eval is asymmetric — you can't easily test bad ideas without harming users.
For product teams
Critical for real product decisions. Offline evals get you 80% of the way, but online results tell you whether users actually care.
For engineers
Instrument logging, run A/B tests, compute metrics on real traffic, implement guardrails to catch regressions.
Related
- Complements offline evaluation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome