A Kubernetes cluster running 40 services can expose tens of thousands of Prometheus metric series before a single alert rule evaluates. Add container logs, OpenTelemetry spans, deployment events, cloud provider metrics, and paging notifications, and the hard problem is no longer collecting evidence. It is deciding which pieces of evidence should change the diagnosis.
That distinction shapes how we built Cordant. Cordant watches production systems and diagnoses incidents when they start, proposing and applying a fix so on-call engineers are woken only when a human decision is genuinely required. Signal ingestion is therefore useful only when it reduces the downstream decision burden. A larger event stream is not automatically a better operating system for production.
My background is in distributed systems and observability. I have built autonomous remediation pipelines for large-scale cloud services and spent a great deal of time inside Prometheus, OpenTelemetry, and Kubernetes internals. In those systems, signal quality depends less on any single metric than on the relationships among metrics, traces, logs, topology, and change history.
That is the design problem this article addresses: what Cordant ingests, how it samples without discarding useful evidence, and how it correlates signals without turning correlation itself into another source of noise.
Start with decision value, not source count
Production systems emit signals for different reasons. A Prometheus counter can show that a failure rate is increasing. A histogram can show that the tail of a latency distribution is degrading. A trace can connect a slow request to a downstream dependency. A Kubernetes event can show that a pod was rescheduled. A GitHub deployment can establish that code changed shortly before the symptoms appeared.
Those signals have different costs and different diagnostic value. High-cardinality labels can make a metric expensive to query. Raw logs can contain rich context, but their volume and inconsistent structure make them difficult to compare over time. Traces provide causal paths, but tail-based retention and sampling decisions affect which paths remain available. A deployment event is sparse, but it can sharply change the probability of a regression hypothesis.
The point is not to collect more signals; it is to collect the signals that change the probability of the correct diagnosis.
This principle gives ingestion a purpose beyond storage. Cordant builds a working representation of the system from the evidence available in the team’s existing tools. That can include Prometheus and Grafana, Datadog, AWS CloudWatch, GCP Monitoring, Kubernetes, and OpenTelemetry sources. It can also include PagerDuty, Opsgenie, Slack, and GitHub, because an incident’s operational context often exists outside the metrics backend.
We do not treat every source as interchangeable. A page from PagerDuty is an assertion that a configured condition crossed an operational boundary. A span is an observation about one request path. A Slack message may be an unstructured report from an engineer. Each gets a different reliability and freshness profile before it contributes to a diagnosis.
An ingestion path that preserves meaning
The first stage is normalization. Cordant maps source-specific records into a common event model while retaining the original dimensions needed for investigation. For metrics, that includes the metric name, label set, timestamp, aggregation type, unit, and source query context. For traces, it includes service and operation identity, duration, status, parent-child relationships, and exemplars when they connect a metric sample to a trace.
Normalization is not flattening. Flattening would discard the distinctions that make a signal useful. A P99 latency observation should not be treated as equivalent to an average. A counter increase should not be interpreted as a current level. A Kubernetes restart count should retain its relationship to the pod, owner, node, namespace, and container. The normalized representation must preserve those semantics so later reasoning does not confuse unlike measurements.
The ingestion path also records freshness and provenance. A delayed CloudWatch datapoint should not receive the same weight as a current scrape. A metric that disappeared because a target was removed should not be interpreted as a healthy zero. An alert that has been firing for six hours carries different information from an alert that crossed its threshold 30 seconds ago.
For each signal, Cordant tracks at least four practical questions:
- What system or component produced it?
- What time window does it describe?
- Which entities and dimensions does it identify?
- What kind of claim does the signal support?
That last question matters. A log line containing “timeout” is evidence of a symptom. A trace showing repeated waits on a database call is evidence about a path. A deployment event is evidence about a possible cause. The ingestion layer keeps those categories distinct instead of allowing a frequent symptom to overwhelm a less frequent but more discriminating event.
Sampling without erasing the incident
Sampling is unavoidable at production scale. Retaining every span, log line, and high-cardinality time series at full resolution can be expensive and can make analysis slower rather than faster. The engineering question is not whether to sample. It is where to sample, what to retain, and how to increase resolution when the system detects that an incident may be forming.
Metrics usually arrive as already aggregated observations. Cordant can preserve the normal scrape or query interval for baseline analysis while requesting finer-grained views around a change point. It also keeps the distinction between raw samples and derived features such as rate, deviation from a recent baseline, error budget burn rate, or a change in histogram bucket distribution.
Trace sampling needs a different treatment. Uniform head sampling is useful for understanding ordinary traffic, but it can underrepresent rare failures. Tail-oriented selection is more useful during diagnosis because traces with errors, long durations, unusual routes, or distinctive dependency paths carry more information. Exemplars provide another bridge: when a metric anomaly points to a trace identifier, the trace can explain what the aggregate measurement cannot.
Logs are sampled by structure and event value rather than by line count alone. Repeated messages from the same pod may add little after the first examples establish the pattern. A new error signature, a change in stack location, or a message that includes a request identifier may deserve retention. Sampling must also preserve a representative slice of normal traffic, otherwise the system loses the baseline required to distinguish a real change from an always-present failure mode.
Adaptive sampling introduces complexity. A sampler that increases retention only after an alert fires may react too late. One that reacts to every small deviation can increase cost and amplify noise. Cordant uses changes in multiple dimensions, such as error rate, latency shape, dependency behavior, and deployment activity, to decide when more evidence is worth obtaining.
There is a real failure mode here: sampling can lose the one request, log line, or label combination that explains a rare incident. No sampling policy guarantees that every important event will be retained. For that reason, Cordant does not treat sampled data as complete history. It tracks coverage, confidence, and missingness, and it lowers diagnostic confidence when the available evidence cannot support a strong conclusion.
Correlation should reduce the search space
Correlation is often described as joining events that occur near one another in time. Time proximity is useful, but it is not enough. A busy service can have thousands of events in the same five-minute window, and many of them will be unrelated. Useful correlation combines time with topology, identity, causal path, and change behavior.
Cordant maintains a topology graph from the relationships visible in Kubernetes metadata, trace edges, service configuration, and observed network calls. It can associate a deployment with the workload it changed, a workload with the pods it owns, a pod with its node and namespace, and a request failure with the downstream service that handled the relevant span. This graph narrows the candidate set before the system compares signal values.
The correlation window is also dynamic. A fast application regression may become visible in seconds. A queue backlog or storage saturation pattern may develop over several minutes. A certificate or credential issue may recur at a fixed interval. Cordant uses the behavior of the signal to choose a useful window rather than applying one universal interval to every incident.
Correlation reduces noise when it collapses multiple observations into one supported hypothesis. For example, a page for elevated 5xx responses, a P99 latency increase, and a cluster of failed traces may represent one incident, not three independent problems. They should produce one diagnosis with supporting evidence and a confidence level, not three competing notifications.
The opposite can also happen. Two anomalies may share a time window but have different causes. A database slowdown and a separate deployment regression can overlap. The topology graph, affected entity sets, and trace paths help keep those incidents separate when their evidence does not converge.
A concrete correlation path
Consider a B2B SaaS platform running 40 microservices on GKE. The platform exposes Prometheus metrics, sends application traces through OpenTelemetry, exports Kubernetes events, and records deployments in GitHub. Its checkout service begins showing a rise in P99 latency, followed five minutes later by a 5xx alert.
Cordant checks whether the symptoms share affected entities and timing. Slow traces are concentrated in checkout requests calling the pricing service. Pricing spans show increased wait time on a database operation, and checkout errors are mostly deadline exceeded responses from that dependency. Kubernetes reports no node pressure, pod restarts are normal, and network errors have not changed.
A GitHub deployment changed the pricing service 11 minutes before the first latency deviation. The deployment altered a query path, while the database connection pool metric shows saturation shortly afterward. The deployment event raises the probability of a regression, while the trace path, pool saturation, query latency, and service errors provide independent support. Cordant can group the visible alerts into one incident, identify the pricing deployment as a leading cause, and propose a rollback. If policy permits automated rollback, it can apply that fix. If rollback requires judgment, it wakes the on-call engineer with the relevant evidence.
Where correlation stops
Correlation is not causality, and it does not eliminate ambiguity. A deployment near an incident may be coincidental, a shared dependency can create symptoms across unrelated products, instrumentation can change, and a trace may omit the critical path because of sampling or propagation gaps.
Cordant therefore treats diagnosis as a confidence-weighted decision process. It can identify competing hypotheses, show which signals support each one, and escalate when evidence is insufficient or the proposed action carries material risk. The system does not guarantee automatic diagnosis, retention of every rare event, or resolution of every underlying problem. In our design-partner deployments, the useful measure is whether the on-call engineer receives a smaller, better-supported decision set.
If you're instrumenting production systems and finding that more signals means more noise, not less, Cordant is built to help turn those signals into a narrower path from detection to diagnosis.
Jordan Kessler is CTO & Co-Founder of Cordant. His work spans distributed systems, observability, and autonomous remediation pipelines for large-scale cloud services, with deep experience in Prometheus, OpenTelemetry, and Kubernetes internals.