Shadow testing
Running a new model in parallel on real traffic without showing results to users, just to see how it would perform.
Think of it like
A stunt double doing the same scene as the actor — you see if they'd succeed without affecting the movie.
Example
You deploy a new ranking model that runs on every search query but doesn't affect results. You log how it would rank things, then later compare against what the old model did.
How it actually works
Shadow testing lets you evaluate on 100% of real traffic without risk. The catch: users don't interact with shadow results, so you can't measure engagement. And running both models costs resources. It's best for measuring offline metrics (relevance, retrieval quality) that don't need user interaction.
For product teams
Safe way to test radical changes. Cost-benefit depends on your traffic and how expensive running two models is.
For engineers
Run both models, log both sets of results, compute metrics on shadow results post-hoc. Skip user interaction data.
Related
- Precursor to canary evaluation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome