At 02:47 AM, an autonomous remediation agent detected a payment-service error spike, reverted one configuration change, and restored healthy request rates in 38 seconds. That sequence is useful only if every action was bounded before the incident began. A system that can diagnose a failure but cannot distinguish a safe rollback from a dangerous intervention is not ready to operate unattended in production.
Autonomous remediation sits at a fork. One path is autonomous resolution: correlate signals, identify a known failure mode, apply a constrained fix, and verify recovery. The other path is human escalation: stop before action when the evidence, scope, or consequence exceeds the agent's authority. Production safety depends less on choosing autonomy everywhere than on making that fork explicit.
Cordant watches production systems, diagnoses incidents as they start, and proposes or applies a fix so on-call engineers are woken only when a human decision is genuinely required. The hard engineering work is not making an agent take action. It is defining the conditions under which action is allowed, measuring whether the result is safe, and making refusal a successful outcome when the conditions are not met.
The fork is a policy decision
An autonomous SRE agent should not begin with a general instruction such as “fix the incident.” That instruction has no useful action boundary. It needs a policy that maps a detected condition to a small set of permitted operations, each with a known blast radius and a verification plan.
A policy might permit a config revert when a recently changed value correlates with a narrow error increase, the previous version is available, and the affected service has enough healthy capacity to tolerate the rollback. The same policy might prohibit a database schema change, a deletion, a cross-region failover, or an alteration to an authentication system. These are not arbitrary limitations. They reflect the cost of being wrong.
The agent also needs a clear distinction between diagnosis and authority. It can collect Prometheus metrics, inspect OpenTelemetry traces, correlate deployment events, and compare Kubernetes workloads against their recent history. That evidence can support a conclusion without granting permission to modify every component mentioned in the incident.
This is not about removing the human from production; it is about reserving human judgment for the decisions that genuinely need it. The practical version of that principle is a fork in the incident lifecycle. Every remediation path should end either in a bounded action with verification or in a human escalation with enough context to make the next decision quickly.
Build the safety envelope before the playbook
A remediation playbook is safe only inside a safety envelope. The envelope describes where the action may run, what it may touch, how long it may continue, and which signals can cancel it. Without those limits, a correct playbook can become an uncontrolled automation loop.
Four dimensions are particularly important.
- Action scope: Name the exact resource types and operations allowed. “Restart the service” is broad. “Restart up to two unhealthy pods in the payments namespace, one at a time” is testable.
- Blast radius: Set limits for replicas, regions, tenants, partitions, or traffic percentage. A scale-out action should have a ceiling, and a rollout pause should not silently become a fleet-wide rollback.
- Temporal bounds: Define a start condition, a maximum duration, and a cooldown. A remediation that keeps retrying after its evidence has gone stale is a new incident.
- Verification: Specify the signals that must improve and the signals that must remain within bounds. Error rate alone is not enough if latency, saturation, queue age, or business transaction success is deteriorating.
These constraints should be enforced outside the agent's reasoning process where possible. Kubernetes admission controls, cloud IAM permissions, deployment tooling, and policy checks can make an unsafe action impossible rather than merely discouraged. The agent can select among permitted operations, but it should not be able to grant itself broader access.
Reversibility matters too. A config revert is safer when the prior configuration is immutable, identified by version, and available for a second rollback if the first change was not the cause. A pod restart is safer when the workload has readiness checks and sufficient replicas. A traffic shift is safer when the original routing state is recorded and restoration is a single bounded operation.
A payment incident with two legitimate outcomes
Consider a fintech platform running a payment processing service on EKS. A deployment changes connection-pool settings. Twelve minutes later, Prometheus shows rising upstream timeouts, OpenTelemetry traces show longer spans at the payment authorization boundary, and queue age begins to climb. The service is still accepting some traffic, so the incident is serious but not yet a total outage.
An autonomous agent correlates the timing, identifies the changed configuration, and compares the current behavior with the previous version. The policy allows a revert if the error rate is concentrated in the changed service, no active migration is using the setting, and the revert affects only that deployment. The agent pauses the rollout, restores the prior configuration, and observes the service for a fixed interval.
Verification is not a single green dashboard. The agent checks authorization success, upstream timeout rate, p95 latency, queue age, pod readiness, and saturation. If those signals move toward their expected ranges without creating a new alert, the action can be marked successful. The incident record should include the evidence, the exact change, the command or API operation used, and the before and after measurements.
Now change one fact. The same symptoms appear, but a database migration is running and the configuration controls connection behavior shared by two payment paths. The prior setting is available, yet reverting it could increase load on the database during the migration. The diagnosis may still be likely, but the action is no longer inside the safety envelope. The correct result is escalation, not a more aggressive guess.
That distinction is central. Autonomy is not measured by how often an agent changes production. It is measured by whether it takes the right branch under uncertainty. In the second case, a useful escalation would state the suspected cause, the conflicting condition, the possible impact of a revert, the relevant dashboards, and the decision that requires an owner.
Stop and escalate is a designed trigger
“Stop and escalate” should be a concrete trigger, not a vague fallback for when the agent feels uncertain. The trigger can be based on evidence quality, action risk, scope, or failed verification.
- Escalate when multiple plausible causes remain and the permitted action would affect more than one of them.
- Escalate when the proposed fix crosses a resource, tenant, region, or data boundary that the policy does not explicitly cover.
- Escalate when rollback state is missing, inconsistent, or older than the evidence supporting the diagnosis.
- Escalate when the first bounded action fails to improve the required signals within its observation window.
- Escalate when remediation would trade an observable availability problem for a less visible integrity, security, or data-loss risk.
- Escalate when the agent detects conflicting signals, such as lower error rate paired with rising queue age or declining transaction success.
A good trigger also prevents escalation storms. If a remediation is blocked, the agent should record that decision and suppress repeated attempts until a material condition changes. It should group related alerts, preserve the causal timeline, and route one actionable page rather than waking several engineers with partial explanations.
The human handoff should carry more than “automation stopped.” It should include the incident start time, correlated signals, recent changes, actions considered, actions ruled out, policy constraints, current blast radius, and the next decision point. That turns escalation into a compressed investigation instead of a reset to zero.
Autonomy has real limitations
There is a temptation to treat bounded actions as a complete answer to production risk. They are not. Observability can be incomplete, telemetry can be delayed, and the most important failure mode may be outside the system being inspected. A service can report healthy while a downstream business process is quietly losing events. A rollback can improve infrastructure metrics while preserving corrupted state.
Distributed systems make causal inference especially difficult. Correlation between a deployment and an incident is useful evidence, but it is not proof. A regional network event, a dependency degradation, or a traffic mix change can overlap the same time window. An agent that overweights recency will repeatedly revert the wrong component.
There is also an operational cost to excessive caution. If every action requires several approvals, autonomy becomes a slower alerting interface. Safety controls must therefore be calibrated with replay, fault injection, and review of real incident timelines. Teams need to learn which actions are genuinely reversible, which signals predict recovery, and which “safe” operations have hidden dependencies.
Human review remains necessary for new failure modes, ambiguous ownership, policy changes, and incidents where business context matters more than infrastructure state. An agent should not decide whether to disable a fraud control, discard a backlog, bypass an authorization check, or alter retention behavior merely because those actions could reduce a metric. Those decisions belong to an accountable owner with the relevant context.
Make the boundary observable
Teams adopting autonomous remediation should monitor the automation itself. Track how often each playbook is proposed, approved, applied, blocked, rolled back, or escalated. Measure time from detection to diagnosis, diagnosis to action, and action to verified recovery. Compare those paths with incidents handled manually, but do not reduce success to MTTR alone.
Useful review questions include: Did the action stay inside its declared scope? Were the verification signals sufficient? Did the agent stop at the correct point? Did the escalation contain enough information? Did the remediation hide a secondary failure? Did the policy make the safe action possible without granting unnecessary permissions?
These reviews should update the envelope, not just the playbook. A repeated escalation may indicate that an action needs better telemetry, a clearer owner, or a narrower policy. A repeated false remediation may indicate that the diagnosis depends on a signal with poor cardinality or delayed delivery. Safety is an operating discipline that changes as the system changes.
Cordant is built around this boundary: diagnose quickly, act only within explicit authority, verify the result, and stop when the evidence or consequence exceeds that authority. For teams evaluating autonomous remediation for production, the useful question is not whether an agent can take action. It is whether you can clearly define the fork, enforce both paths, and trust the agent to choose escalation when production safety requires it.