Alert fatigue is usually diagnosed as a failure of discipline, but the deeper failure is often architectural: the people receiving alerts do not control the systems that decide what deserves an interruption. Telling engineers to improve their signal-to-noise ratio assumes they own the alerting pipeline, the routing policies, the service-level objectives, and the operational context behind every notification. Most do not.
When an on-call engineer receives a page for a symptom they cannot investigate, a threshold they cannot change, or an event that another system already resolved, the problem is not simply that the team needs to tune its alerts. The organization has placed responsibility for response at the edge of a system whose design is controlled elsewhere.
The alert is the last stage of a longer system
A page looks like a local event. An engineer sees a notification, opens a dashboard, and decides what to do next. But the page is the output of several upstream choices:
- Which telemetry is collected and retained
- Which conditions become Prometheus alerting rules or equivalent checks
- Whether thresholds represent symptoms, causes, or mere activity
- How alerts are grouped, deduplicated, inhibited, and routed
- Whether the receiving team has the context and permissions to act
- Whether the alert reflects an SLO violation or only an internal system state
By the time an alert reaches PagerDuty, much of its meaning has already been determined. The application team may own the service, but not the shared exporters, the monitoring configuration, the notification integration, or the escalation policy. It may be asked to reduce noise without having authority over the mechanisms producing that noise.
This is why alert fatigue can persist even after individual teams make reasonable local improvements. A team can remove duplicate rules while a platform service continues to emit low-value infrastructure alerts. It can rewrite descriptions while routing policies continue to page the wrong group. It can adopt SLO-driven alerting while a separate process pages on every elevated CPU reading.
The alert is not the system. It is the visible edge of the system.
Who owns the alerting layer?
Many organizations have a clear answer to the question, "Who is on call for this service?" They have a less clear answer to, "Who is accountable for whether this page should exist?" Those are different responsibilities.
Service teams usually own the behavior of their applications. Platform engineering often owns the monitoring stack, shared dashboards, telemetry collection, and the mechanisms for delivery. Security, database, and infrastructure teams may add their own checks. A central operations group may define PagerDuty routing policies. Each layer can be locally reasonable while the complete alert taxonomy becomes incoherent.
Without explicit ownership, alerting becomes a collection of inherited assumptions. A critical alert from one team may mean "customer impact is occurring." A critical alert from another may mean "someone should inspect this during business hours." Both arrive through the same paging channel. The recipient must infer urgency from the content of the message, often while already handling another incident.
Platform teams are sometimes told to standardize alerts by enforcing naming conventions and severity labels. Those controls help, but taxonomy is not just vocabulary. It is a contract about action. A page should identify an operational condition, the team responsible for it, the likely scope of impact, and the first useful investigative path. If those fields are absent, the platform has not produced a reliable signal. It has produced an interruption.
Ownership should therefore include more than the right to edit a rule. It should include responsibility for the lifecycle of the signal: creation, review, routing, suppression, escalation, and retirement. An alert that no longer leads to a distinct action is operational debt, even if its rule is technically correct.
A scenario that looks like a tuning problem
Consider a platform team managing alerting for 12 engineering teams. The platform team operates Prometheus, maintains shared recording rules, and manages the PagerDuty integration. Each product team owns its service and receives pages for production incidents.
One product team reports that its engineers are getting paged several times a week for elevated request latency. The latency alerts fire during routine traffic shifts, clear within minutes, and rarely correspond to a customer-visible SLO violation. The team is told to tune its thresholds.
That request sounds reasonable until the ownership boundaries are made explicit. The product team does not own the histogram configuration, which determines how latency is measured. It does not own the shared recording rule that aggregates latency across regions. It cannot change the grouping key in PagerDuty, and it has no authority to alter the platform-wide evaluation interval. Its engineers can adjust a service-specific threshold, but doing so may hide a real problem in one region or make the rule inconsistent with other services.
The team lowers the alert sensitivity. A month later, an actual latency incident is detected later than expected. The organization concludes that the team tuned its alerts incorrectly. In reality, the original request asked a service team to compensate for a cross-layer design problem.
A better review would ask several questions before changing the threshold:
- Is the alert measuring a symptom that affects users, or an infrastructure condition that may be harmless?
- Does the query preserve the dimensions needed to identify the affected service and region?
- Does the notification route to the team that can mitigate the condition?
- Is the evaluation window long enough to distinguish a transient fluctuation from sustained impact?
- What action is the on-call engineer expected to take when the page fires?
If those questions have no clear answers, changing the threshold is not alert hygiene. It is moving uncertainty from one part of the system to another.
What this argument is, and what it is not
This is not an argument that service teams have no responsibility for their alerts. Teams should understand their golden signals, define meaningful SLOs, review their alert rules, and remove pages that do not lead to action. A platform architecture does not excuse careless configuration.
It is also not an argument for centralizing every alert decision. A central platform team cannot understand every application workflow or determine the right customer-impact threshold for every service. The people closest to the service need authority to define its operational signals.
The boundary is this: local teams should own the meaning of their service alerts, while platform teams should own the conditions that affect how those alerts are generated, enriched, routed, and acted upon at scale. Neither group can solve alert fatigue alone. The interface between them needs explicit contracts.
Those contracts can be lightweight. A platform may require every paging rule to include an owning service, an SLO or user-impact rationale, a runbook link, and a defined response. It may provide standard recording rules and routing primitives while allowing teams to set service-specific objectives. It may review the number of pages generated by a rule, not as a punitive score, but as evidence that the rule deserves examination.
The goal is not fewer alerts at any cost. The goal is that an alert reliably changes what the recipient does next.
The useful counterpoint: not every page needs perfect context
There is a practical counterpoint to this structural view. During an active incident, an imperfect alert can still be valuable. A page about elevated error rates may arrive before a clean SLO burn-rate alert is available. A noisy dependency alert may provide the first indication that a wider failure is developing. Waiting for a complete taxonomy, perfect ownership metadata, and a fully automated diagnosis can delay detection.
That nuance matters. Alert quality is not binary, and platform design should not become a reason to postpone useful signals. Early warnings have a place, especially when they are clearly labeled, routed to a team that can investigate, and separated from pages that require immediate intervention.
The problem begins when provisional signals acquire the same urgency as confirmed customer impact and remain in production indefinitely. A warning can be useful without being a page. A page can be technically accurate without being actionable. The operational system needs to distinguish those states.
This is where review practices matter. After an incident, teams should examine not only which alerts fired, but which alerts were ignored, duplicated, late, misrouted, or missing. That review should include the platform layer. If the same class of noise appears across multiple services, it is evidence of a shared design issue rather than a series of unrelated team failures.
Designing for decisions, not notifications
A healthier alerting architecture starts with the decision an engineer may need to make. If the decision is to roll back a deployment, the signal should connect recent change activity with a meaningful service symptom. If the decision is to fail over a dependency, the alert should provide evidence that the dependency is affecting the service and identify the available mitigation. If no human action is expected, the event may belong in a dashboard, log, ticket, or automated remediation path instead of the paging channel.
That last category deserves more attention. Some incidents follow recognizable patterns: a process crashes, a queue backs up, a certificate approaches expiration, or a known dependency returns a repeatable failure. When diagnosis and remediation are sufficiently bounded, software can investigate and apply the approved fix while preserving an audit trail. The human should be brought in when judgment, authorization, or an unusual tradeoff is required, not merely because a metric crossed a threshold.
This does not eliminate the need for on-call engineers. It makes their attention more specific. The platform should filter routine, diagnosable conditions and provide the context needed for the incidents that remain. Cordant is built around that operating model: watching production systems, diagnosing incidents when they start, and proposing or applying a fix so an engineer is woken only when a human decision is genuinely required.
Alert fatigue is therefore a useful test of platform design. Ask who can change the alert, who can interpret it, who can act on it, and who is accountable when it becomes noise. If the answers point to different teams with no shared contract, asking the on-call engineer to "be more discerning" is not a remedy. It is a sign that the system has assigned the final responsibility without granting control over the conditions that create it.
The next time a team reports too many pages, start by tracing one alert from telemetry to notification to response. The path usually reveals whether the organization has a tuning problem at the edge or an ownership problem throughout the system.