Distribution shift
When your model's test data comes from a different distribution than what it'll see in production.
Think of it like
Training a classifier on daytime photos, then using it on nighttime photos.
Example
Your NER model is trained on news articles. You deploy it to social media. Tweets are informal, have slang, use emojis — the distribution shifted. Performance drops.
How it actually works
Distribution shift is subtle because performance on old data stays good — you don't notice until production fails. Types include covariate shift (input distribution changes), label shift (class distribution changes), and concept drift (the relationship itself changes). Some shifts are fixable (retraining), others require architectural changes.
For product teams
Expect model performance to degrade post-launch. Monitor for it. Plan retraining cycles.
For engineers
Collect examples from production. Compare feature distributions to training. Set up drift detection.
Related
- Real-world problem.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome