Alert fatigue is a signal design problem, not a people problem. Post-mortems repeatedly identify it as a contributing factor. Engineers respond slowly to alerts that have been noisy for months. When a real incident arrives on that channel, the delay lets the damage spread further than necessary.
Organizations usually respond with more training, clearer escalation paths, and improved on-call procedures. Some add a quarterly alert-quality review. These steps miss the root issue: this is not a process problem. It is an engineering problem.
Signals, Not Symptoms
Most alerting relies on metric thresholds, which are easy to apply incorrectly. The usual error is paging on internal metrics that correlate with trouble instead of external symptoms showing that trouble exists or is imminent.
CPU utilization above 80% is an internal metric. It can correlate with trouble, yet a CPU-constrained system may work perfectly from the user's perspective. An alert firing whenever CPU reaches 82% during a traffic spike teaches engineers to treat it as background noise.
Request latency above the p99 SLO threshold is an external symptom. When it fires, users are receiving slow responses. That nearly always merits engineer attention. The signal is immediately actionable and directly connected to user impact.
The principle is simple: "alert on symptoms, not causes." Apply it by reviewing every alert with one question: if it fires while users experience no degraded service, should it have fired at all?
Why Alert Systems Keep Accumulating
Alerting systems usually grow, never shrink. If a failure occurs without an alert, an engineer adds one. If an alert lacks context, related alerts are added for diagnosis. When a service is deployed, its health metrics join the same alerting system, using the same thresholds as the rest.
No one deliberately creates noisy alerts. Noise builds through individually defensible additions. After two years, a system that began with 20 alerts may have 180, and no one understands why each exists.
The remedy needs a recurring deletion cycle, not only an addition cycle. A practical rule: an alert firing more than N times per week without ever producing a runbook action should be considered for deletion. It is not calibrated to the system's real failure modes.
We use a 90-day window and ask whether no documented action was taken on more than 80% of firings. This finds alerts that have lost relevance while preserving rare alerts that matter when they fire.
Hysteresis and Aggregation
Two technical patterns cut alert noise without weakening detection: hysteresis and aggregation.
Hysteresis filters out brief transient spikes. An alert firing after CPU exceeds 80% for one 1-minute polling interval will fire constantly under variable load. Requiring CPU to exceed 80% for 5 consecutive minutes limits firing to sustained conditions, when they indicate a real problem. Detection sensitivity is nearly identical, while noise falls sharply.
Aggregation handles failure fan-out from upstream services. One upstream failure can trigger dozens of dependent-service alerts at once. If authentication goes down, every dependent service may fire its timeout alert. These are not separate failures. They describe one incident with one root cause.
Good alerting infrastructure detects alerts likely tied to one upstream event and combines them into a notification naming the probable root service. Engineers handle one incident instead of 17 simultaneous pages describing the same underlying failure.
Alert Volume Versus Alert Value
Counterintuitively, fewer alerts do not necessarily mean poorer detection coverage. Lowering alert volume often improves detection quality by raising the signal-to-noise ratio of the alerts that remain.
Comprehensive monitoring is not the problem, and neither is collecting fewer metrics. The interrupting set, especially at 3am, should be far smaller than the set of conditions you want to observe. Use dashboards and logs for diagnosis. Use alerts for conditions that need immediate human action.
For every proposed alert, ask: if this fires, what specific action will the on-call engineer take within the next 15 minutes? "Investigate and see" means it is premature. Set completion criteria for the investigation and map each result to its consequence. Then alert on the outcome requiring action, not the precondition that starts investigation.
What Autonomous Detection Changes
Autonomous detection systems like what we are building at Cordant change the model. Instead of depending entirely on human-defined thresholds, they can correlate multiple signals at once and identify anomaly patterns without requiring every threshold to be defined in advance.
Well-designed alerts are still necessary. Human-defined alerts for critical SLO thresholds remain an important direct safety net. But autonomous correlation reduces the pressure to put diagnostic alerts on the paging channel. The system can perform that correlation instead of waking an engineer at 2am to interpret four simultaneous metric deviations.
The principle does not depend on tooling. Solve alert fatigue upstream through signal design, not downstream with training or process. On-call engineers respond to the signal quality they receive: high-value signals get a strong response, while enough noise eventually makes noise and signal indistinguishable.