Partial observability was the hardest engineering problem we faced while building Cordant. We started with a simple model: an agent watches metrics, matches patterns, and runs the right runbook. We expected implementation to be easy because we knew distributed systems, observability pipelines, and reliable automation.
That model was roughly right, yet of little use for the real problem. The difficult work was not monitoring or runbook execution. It was deciding what to do with incomplete information and defining the safety rules for uncertain decisions.
Partial Information Problem
Production incidents rarely come with complete, clean diagnostic data. Metrics lag, traces disappear under load, and dependent services report symptoms at the same time. That makes cause and consequence difficult to separate. An agent that waits for certainty will act too slowly or send too many incidents to people.
We first responded by adding sources: more metrics, trace samples, and log correlations. That did not fix the issue. It produced more incomplete, sometimes conflicting signals. Under time pressure, reconciling 15 partial sources is not better than using 5 carefully selected ones. The agent is simply slower and less certain.
The useful pattern was tiered confidence with explicit uncertainty propagation. For every root-cause hypothesis, the agent keeps a confidence score based on available evidence. High-confidence hypotheses move to action selection. Low-confidence ones go to humans with the partial evidence included.
The agent also states exactly which evidence is absent. Instead of saying, "I could not diagnose this," an escalation says, "the latency spike matches a memory pressure scenario, but database-layer memory metrics are unavailable; this is what I can see and what would confirm the hypothesis." That gives the on-call engineer far more to work with than an "unknown error" handoff.
Ambiguous Signal Handling
Some signals are truly ambiguous. One metric pattern may have several root causes. API gateway CPU could reflect a traffic surge, a runaway process, or a dependent-service timeout that exhausts the connection pool. Each requires a different remedy, and choosing incorrectly can worsen the incident.
The obvious strategy is to choose the likeliest cause and act. It works until it fails, and then the wrong remediation can materially lengthen the incident.
We settled on a pattern we call "safe actions first." When two or more causes remain possible, the agent chooses a remediation that is safe under each one. Scale-out fits: it directly helps a traffic surge, buys time during a runaway process, and may reduce connection-pool pressure when a dependent service is timing out. The agent then checks whether the incident is improving. If not, recovery-period metrics usually clarify the original hypothesis, allowing a second, more targeted action.
This takes longer than acting on the likeliest cause, but it is more dependable. The agent's purpose is not cleverness. It is resolving the incident correctly.
Safe Rollback Challenge
Every remediation can fail in two ways: it may do nothing or make the situation worse. The agent must recognize when to stop following that path, choose another, or escalate.
We first used a simple timeout: if the incident remained unresolved N minutes after an action, try the next action or escalate. That rule was too blunt. Correct remediations can need time to take effect. After scale-out, pods may need 30 to 90 seconds to become ready, so checking at 60 seconds can falsely indicate that the incident remains unresolved.
A better approach uses action-specific resolution windows. Each runbook action defines how long its effect may take, and the agent waits through that window before evaluating again. It then checks the condition tied to that action. If the condition is still unmet, it examines whether the system is improving, indicating progress, or flat or degrading, indicating failure.
A worsening trajectory after an action indicates rollback. If reversible, the agent undoes the action, records what it attempted, and escalates with that context.
Calibrating Team Trust
We did not initially anticipate how important it would be to align confidence thresholds with what the on-call team considers acceptable. Escalating 50% of incidents is not necessarily a job failure, but teams expecting 20% based on the product description will see the difference as failure. Trust declines even when the behavior itself is correct.
We now begin each pilot with an explicit expectations review. We show the incident categories the agent should handle autonomously, those it should escalate, and how confidence thresholds work. We ask teams to flag categories that seem incorrect. This exposes assumptions that would otherwise surface only as disappointed expectations six weeks into deployment.
What We Would Change
Starting again, we would prioritize natural-language explanations sooner. An escalation is the handoff from machine response to human response, so its message matters. We initially optimized for technical accuracy and gave too little attention to presenting that information clearly and usefully.
At 3am, an SRE does not want a JSON dump of internal state. They need a plain statement of what is wrong, what was attempted, the current state, and the agent's recommended next step. Making that communication layer work proved as important as building the diagnostic layer.
A production-grade agent is less about machine learning components than about decisions under uncertainty, action safety constraints, and the design of the human escalation path. These are familiar reliability engineering problems, applied in a new setting.