Alerting
Automated notifications when metrics or logs hit a threshold—latency spike, error rate jump, model drift.
Think of it like
A smoke alarm that wakes you up when there is danger, not you checking the house every minute.
Example
Inference latency p99 > 500ms triggers a PagerDuty alert; on-call engineer sees it within 30 seconds.
How it actually works
Alert fatigue kills response: if 90% of alerts are false positives, the on-call engineer will ignore them. Tune thresholds carefully and use multi-signal rules (not just one metric). Routing rules send the right alert to the right team. Must include runbook links to unblock engineers.
For product teams
Dramatically shortens time-to-incident and time-to-resolution.
For engineers
Rules engine (Prometheus, Datadog, CloudWatch) evaluates conditions on metrics; fires webhook, email, or SMS; integrates with incident management.
Related
- Driven by.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome