Alerting and On-Call

Alerting is the bridge between observability and action. The purpose of an alert is to notify the right person at the right time about a condition that requires human intervention. Done well, alerting catches problems early and reduces incident impact. Done poorly, it leads to alert fatigue, missed issues, and burned-out engineers.

---

Alerting Philosophy

Google's SRE book established foundational principles for alerting:

1. Every alert should be actionable — if no one needs to act, it should not page
2. Alert on symptoms, not causes — alert on "error rate is high" rather than "CPU is at 90%"
3. Alert on SLO burn rate — tie alerts to what matters to users
4. Reduce noise ruthlessly — fewer, better alerts lead to faster response

Rule of thumb: If an alert fires and the on-call engineer says "I can ignore this," the alert should be removed or changed.

---

Alerting Components