Canary evaluation
Rolling out a change to a small fraction of real users and monitoring for problems before going to 100%.
Think of it like
A canary in a coal mine — send in a small probe first to check for danger.
Example
You deploy a new model to 5% of search traffic. If latency stays good and click-through rate doesn't drop, roll to 10%. If anything breaks, rollback immediately.
How it actually works
Canary deployments reduce risk but require automated monitoring and quick rollback. You need alarms for latency, errors, and engagement metrics. The cost of rollback must be lower than the cost of breaking all users. And you need enough traffic in the canary to detect real problems (5% might miss low-frequency bugs).
For product teams
Standard practice for high-stakes changes. Non-negotiable for production services.
For engineers
Implement traffic splitting, automated metric collection, and rollback triggers. Monitor continuously.
Related
- Production-safe eval pattern.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome