At 2:17 p.m. on a Tuesday, a checkout service crossed its latency threshold after a routine configuration change. The alert routed to the primary on-call, who checked the dashboard, compared recent deployments, found the change, rolled it back, and posted an incident note before the error budget burn rate returned to normal. The incident lasted 11 minutes, but the engineer lost most of an hour to triage, coordination, and the recovery record.
Nothing about that response was careless. The alert was valid, the runbook was accurate, and the engineer followed the escalation ladder correctly. Yet the organization still spent a human hour proving that a known class of failure had occurred and applying a reversible fix. That observation points to a deeper source of SRE toil: not only how many alerts arrive, but the assumption that every alert must first become a human task.
Toil begins in the response model
Toil is usually measured through visible symptoms: pages per shift, repetitive tickets, manual deploy steps, or the percentage of incidents that consume engineering time. Those measures are useful, but they can lead a program toward local improvements while leaving the response model untouched.
A team may tune alert thresholds, consolidate dashboards, and automate a few runbook commands. The incident triage queue becomes somewhat quieter. The remaining alerts still follow the same path, however: signal, page, human investigation, human decision, manual or scripted remediation, and post-incident documentation. Every event still waits for a person to establish what happened and decide what to do next.
That sequence creates toil even when individual alerts are well designed. A healthy service can generate a valid alert because a dependency has slowed, a worker pool has exhausted its capacity, or a rollout has introduced a narrow regression. The alert is not noise. The human dispatch is the expensive part when the condition is recognizable, the remediation is bounded, and the risk of acting is understood.
Reactive SRE is often described as an alerting problem because alert volume is easy to count. From a first-principles perspective, the more important question is: what work must remain with a human? If diagnosis consists of correlating timestamps, recent changes, logs, metrics, traces, and known failure patterns, that work may be suitable for automation. If the next step is a documented rollback or a controlled capacity adjustment, applying the fix may also be suitable for automation. The human should enter when judgment, authorization, or an unusual tradeoff is genuinely required.
Reactive dispatch turns incidents into a queue
On-call systems are designed around interruption. Prometheus or Datadog detects a condition, an alert router evaluates ownership, PagerDuty starts an escalation ladder, and an engineer opens the incident channel. That chain is valuable for failures that require immediate attention. It also creates a queueing system in which the person with the pager becomes the default processing layer for production signals.
At low volume, the cost can look acceptable. At higher volume, small incidents compete with one another. A latency alert interrupts an engineer investigating a memory leak. A transient dependency failure creates a second page before the first has been acknowledged. A low-severity alert waits in Slack while the on-call handles a customer-impacting outage. Context switching adds time that does not appear in the alert duration itself.
Reactive dispatch also makes prioritization harder than it looks. Severity labels are assigned before the full context is known. A page may be urgent because the symptom is severe, or it may be harmless because a redundancy mechanism is working as designed. Conversely, a small error-rate increase may be the first visible signal of an event that will consume the error budget later in the day.
Humans can make these distinctions, but requiring a human to make every distinction is a design choice. It treats correlation and first response as scarce expert labor. The result is a system where the escalation ladder absorbs work that could have been resolved before an engineer was woken.
We are not saying reactive SRE is incompetent; we are saying the response model itself guarantees toil at a certain scale. Even an excellent team with clear ownership, disciplined incident command, and thoughtful runbooks will accumulate repetitive work if production signals always wait for a person to interpret them.
What a proactive SRE model changes
Proactive SRE does not mean predicting every failure or eliminating the on-call rotation. It changes where analysis begins and who performs the first pass. A proactive model continuously watches for relationships between signals, recent changes, service dependencies, and known operating conditions. It asks whether several weak indicators describe one incident, whether a symptom matches a known failure mode, and whether a safe remediation is available.
That approach moves the system from reactive dispatch to proactive signal correlation. Instead of paging on every isolated condition, the system can assemble an incident narrative while the event is developing. It can identify that a deployment preceded a rise in latency, that only one region is affected, and that rolling back the newest version has succeeded for the same failure pattern before. The important output is not another dashboard. It is a proposed explanation and a bounded next action.
Automation should be graduated rather than binary. A response may begin with observation and evidence collection, continue to a recommendation, and proceed to an approved remediation when the conditions are clear. A rollback can be applied automatically when the affected version, scope, and recovery test are unambiguous. A database failover or a change to traffic policy may require a human decision because the consequences are broader.
For platform teams, this creates a more useful division of labor. Systems handle repetitive detection, correlation, diagnosis, and known-safe actions. On-call engineers handle ambiguous incidents, competing objectives, policy decisions, and changes that carry material risk. The goal is not to remove people from production operations. It is to reserve their attention for work that benefits from experience and judgment.
Toil reduction needs a different measurement
Reducing page volume is a reasonable objective, but it is not enough. A team can lower pages by raising thresholds and create a larger blind spot. It can move work from an on-call engineer to a platform engineer through ticket routing. It can automate the command while leaving diagnosis and approval manual. Each change may improve one metric while preserving the underlying burden.
A more complete toil taxonomy should distinguish at least four categories:
- Detection toil: finding the relevant signal across metrics, logs, traces, and deployment history.
- Diagnosis toil: determining the likely cause and separating correlated symptoms from unrelated noise.
- Execution toil: running a known remediation, verifying its effect, and reverting it if necessary.
- Coordination toil: routing the incident, finding the owner, opening the channel, and recording routine updates.
Post-mortem action items should identify which category consumed time. “Improve alerting” is too broad to guide a durable change. “Automatically correlate deployment events with regional latency alerts” is specific. “Apply the documented rollback when the canary error rate exceeds the defined threshold and health checks recover” is more specific still.
Teams should also measure human wakeups per resolved incident, time spent before a useful diagnosis, percentage of incidents resolved through known actions, and the proportion of pages that required a judgment call. These measures expose whether automation is actually changing the response model or simply decorating the existing one.
SLOs and error budgets provide another useful boundary. A proactive system should not optimize for quiet at the expense of reliability. If it suppresses an alert, it should preserve evidence and make the suppression explainable. If it applies a fix, it should verify the service objective afterward. Reliability work is not successful because fewer people saw the incident. It is successful when the service recovered safely with less unnecessary human effort.
Proactive work has real costs
There is a reasonable counterargument to all of this: proactive systems can create their own toil. Building correlation logic takes time. Runbooks need explicit preconditions. Remediations require permissions, audit trails, rollback paths, and careful testing. A poorly designed automation layer can act on incomplete evidence, amplify an outage, or produce false confidence that an incident is under control.
Those costs are not theoretical. I have seen teams create elaborate health checks that no one trusted because the checks were difficult to interpret. I have also seen “automated” runbooks that still required an engineer to copy identifiers between tools, confirm the scope, and click through several approval steps. The command ran faster, but the cognitive work remained.
Proactive SRE therefore needs operating constraints. Every automated action should have a defined scope, a clear trigger, an observable result, and a stop condition. The system should preserve the evidence used for diagnosis and make its reasoning inspectable during review. Permissions should be narrow, and high-impact actions should remain behind a human approval boundary.
Consider a second scenario. During a weekend shift, a worker service begins restarting because a downstream API has started returning malformed responses. The first few restarts are handled automatically, but the restart loop continues. A proactive system can identify the dependency correlation, stop applying the same restart, and escalate with the relevant logs and request samples. It should not keep retrying simply because restarting has worked in other incidents.
That is where proactive SRE differs from blind self-healing. The system needs to recognize when a familiar remediation no longer improves the service. Human intervention remains necessary for novel failure modes, conflicting signals, security-sensitive events, capacity decisions, and incidents where the safest action is to preserve the current state for investigation.
Changing the habit of on-call
The organizational effect matters as much as the technical one. A reactive culture teaches engineers that availability depends on vigilance. The person who responds fastest becomes the safety net for gaps in ownership, documentation, and automation. Over time, heroic response can become part of the team identity, even while the rotation becomes harder to staff.
A proactive model changes the conversation from “Who is awake?” to “What decision is actually needed?” That shift can improve runbook quality because each procedure must state its assumptions and limits. It can improve post-mortems because the team can distinguish missing automation from missing architectural safeguards. It can also make on-call work more sustainable without treating reduced interruptions as a sign that engineers are less committed.
Adoption should start with a narrow class of incidents. Choose a repetitive failure with a known signal pattern, a bounded remediation, and a reliable verification check. Measure how often the action is appropriate, how often it is stopped, and whether the service objective improves. Expand only after the evidence shows that the automation is safe and understandable.
Cordant is built around this model: it watches production systems, diagnoses incidents as they start, and proposes or applies a fix so on-call engineers are woken only when a human decision is genuinely required. The practical test is not whether an autonomous SRE agent can handle every incident. It is whether it can remove the predictable work that keeps trained engineers in the incident triage queue.
If your toil reduction program is running in place, map each recurring page to the human step it requires. Then ask whether detection, diagnosis, execution, or coordination truly needs a person. That answer is often the starting point for a more proactive response model.