1. Two-strike rule on every alert
Never page on a single failed check. Always require two consecutive failures. This single change eliminates 80 percent of false alarms with under 60 seconds of added detection time.
2. Multi-region consensus
Run every important check from at least three regions. Open an incident only when two or more regions agree the target is down. This handles transient network problems at the probe location.
3. Separate channels for criticality tiers
Critical-severity alerts go to SMS. High-severity to Slack. Medium to email. Low to a daily digest. Mixing them in one channel teaches everyone to ignore the channel.
4. Quiet hours for non-critical rules
Nobody should be paged at 3am for a flaky cron job. Configure quiet hours on every rule below Critical severity.
5. Test alerts on a schedule
Once a quarter, fire a test alert through every channel. Slack webhooks rotate. SMS providers deprecate APIs. Test channels go cold without notice.
6. Escalation chains with named individuals
Critical alerts: tier 1 Slack channel, tier 2 SMS to on-call, tier 3 SMS to engineering manager. The chain must include real names, not aliases that can become empty.
7. Alert on absence, not just failure
A monitor that has not run in 10 minutes is a problem too. Set up heartbeat-style monitoring on the monitors themselves if your tool supports it.
8. Document the runbook in the alert payload
Every alert should link to a one-paragraph runbook: what this means, who to wake up, where the logs live. A 3am responder should not have to figure this out fresh.
9. Status page is part of the alert flow
When an incident opens, the same broadcast should hit your public status page. Customers see the same update at the same time as your team — no support pile-up.
10. Resolve with cause and consequence
Every resolved incident captures two fields: cause (the root) and consequence (what customers experienced). These feed your monthly retrospective and your reliability metrics.
11. Track MTTR by service, not by team
Mean-time-to-recovery should be tracked per service. Cross-team averages hide the services that are getting worse.
12. Less is more — start with five monitors
A wall of green dashboards with 200 monitors creates blindness. Start with five high-signal monitors that cover your customer-facing paths. Add more only when the existing alerts are trusted.