The trace showed a 247 ms latency spike somewhere in the span chain, then went silent. The missing event was not another slow downstream call. It was a node being evicted under Kubernetes memory pressure, with the kernel OOM killer terminating a workload before the request could produce another span.
That distinction matters during an incident. A trace can show the request entering an API, crossing a service boundary, waiting on a database, and returning an error. It may show precisely where time accumulated inside the instrumented path. It usually cannot show the infrastructure event that made the process disappear, changed the node topology, or exhausted the resource that the request depended on.
This is not a failure of tracing. It is a boundary of what tracing records. The trace begins when propagation context reaches an instrumented process. Many production incidents begin earlier, in the scheduler, kubelet, kernel, network fabric, control plane, or deployment system.
The trace starts after the triggering event
Consider a payment service running twelve microservices on Kubernetes. A checkout request enters the edge gateway and propagates a W3C trace context through the payment, inventory, and authorization services. The resulting waterfall shows a normal gateway span, a payment span, and a child span waiting 247 ms on authorization. Then the parent span closes with a timeout.
Looking only at the waterfall, several explanations appear plausible. Authorization may be overloaded. A database query may have regressed. Network latency may have increased. The sampling system may have dropped a child span. None of those hypotheses explains why the authorization pod stopped emitting telemetry at the same moment.
Node metrics reveal that the node crossed its cgroup memory pressure threshold. The kubelet began eviction activity, and the kernel OOM killer terminated a process. The request had already entered the system, so its trace contained the symptoms of the interruption. It did not contain the node condition, the competing workload that consumed memory, or the eviction decision. Those facts existed in Kubernetes events, kubelet logs, cgroup counters, and kernel messages, not in the request's span graph.
A trace is causally downstream from its instrumentation boundary. If the process is paused, killed, isolated from the network, or removed from service discovery, there may be no process left to create the span that would explain why the request failed.
What falls outside the instrumentation boundary
Infrastructure events create gaps in several recurring ways.
- Resource pressure: CPU throttling, memory limits, file descriptor exhaustion, disk saturation, and conntrack depletion can delay or terminate work without producing an application span that names the resource failure.
- Topology changes: node drains, pod rescheduling, availability zone degradation, and service endpoint changes can alter the request path between two trace-producing spans.
- Control plane actions: an autoscaler, deployment controller, admission policy, or kubelet can change what is running while the trace only records the resulting timeout or retry.
- Kernel and network behavior: packet drops, TCP retransmission, connection tracking failures, scheduler delays, and OOM kills occur below most application instrumentation.
- Uninstrumented asynchronous work: queue consumers, batch jobs, cron tasks, and event handlers may cause load or state changes without sharing the original propagation context.
The trace can still expose the effect. A parent span may wait longer. A retry span may multiply traffic. A downstream call may return an unavailable status. What it cannot guarantee is an explanation for every state transition outside the process that emitted the span.
Sampling and propagation make the gap wider
Even events inside the application boundary can disappear from the evidence. Head-based sampling makes the decision near the start of a request, before the system knows whether the request will become interesting. A rare failure may be sampled out because the request looked ordinary at ingress.
Tail-based sampling improves the odds of retaining slow or failed traces, but it does not recover a span that was never emitted. If a process is OOM-killed before export, the collector cannot infer the missing child span. If the collector itself is overloaded, telemetry loss can look similar to application silence. The absence of a span is therefore ambiguous. It may mean the operation did not happen, the process could not report it, the exporter dropped it, or the trace was never selected.
Propagation also breaks at boundaries that are common in real systems. A message may be published without trace context. A queue consumer may create a new root span. A third-party client may strip headers. A retry may use a different context from the original request. A deployment hook or scheduled job may change application state without belonging to any request trace.
These are not arguments for instrumenting every possible surface indiscriminately. They are reasons to treat trace completeness as an operational property with measurable limits. Teams should know which services propagate context, which queues preserve links, which exporters can lose data, and which infrastructure signals are required to interpret a silent span chain.
Tracing is necessary, but not sufficient
Distributed tracing is excellent at what it was designed for: request-path attribution within an instrumented service boundary. The issue is that many production failures start outside that boundary.
Within the boundary, trace data is often the fastest way to answer difficult questions. Which dependency added latency? Did retries fan out? Did a request cross an unexpected service? Is a timeout local to one call or repeated across thousands of requests? Trace exemplars can connect a high-cardinality symptom to a concrete request, while span attributes expose tenant, route, status, and deployment dimensions that aggregate metrics cannot represent as precisely.
The mistake is asking the trace to serve as the complete incident record. A useful investigation correlates the trace with signals that describe the environment around it. Prometheus can show memory pressure, throttling, queue depth, and error rates. Kubernetes events can show scheduling, eviction, and rollout activity. Kubelet and kernel logs can explain process termination. OpenTelemetry collector metrics can reveal telemetry backpressure or export failures. Deployment records can establish whether the topology changed before the first failed request.
The correlation does not need to force infrastructure events into the span tree. A node eviction is not a child span of an HTTP request, and representing it that way can create a misleading causal model. It is often more accurate to preserve separate event streams and correlate them by time, workload, node, namespace, deployment revision, and affected dependency.
Build incident evidence around the trace
For an incident like the 247 ms spike, the operational question is not simply whether a trace exists. It is whether the investigation system can connect the trace's last known state to the surrounding changes.
That requires retaining a small set of high-value relationships:
- the workload, pod, node, and deployment revision associated with each span;
- the first trace timestamp where latency, retries, or errors changed;
- node and cgroup resource conditions before and during that timestamp;
- Kubernetes events involving eviction, rescheduling, readiness, or endpoint changes;
- collector health, sampling decisions, and export errors for the affected period;
- the topology of retries and replacement pods after the original process disappeared.
With that context, a silent trace is evidence rather than a dead end. If the last span belongs to a pod that was killed seconds later, the missing child is no longer mysterious. If every affected trace terminates on one node while other nodes remain healthy, the incident has a different shape from a dependency regression. If traces disappear only from one collector route, the application may be healthy and the observability pipeline may be the component under pressure.
Autonomous incident diagnosis has to make the same distinction. It should not interpret every missing span as an application failure or every timeout as a slow dependency. It needs to inspect the infrastructure timeline around the trace, identify whether the request path was interrupted by a resource or topology event, and separate a broken service from a broken observation path. The proposed action might be a rollback, a workload move, a resource adjustment, or no application change at all.
If distributed tracing is your primary incident investigation tool and you keep hitting infrastructure blind spots, Cordant can help connect request symptoms with the Kubernetes, resource, and deployment events that preceded them, so on-call engineers are involved when a human decision is genuinely required.
Jordan Kessler is CTO & Co-Founder of Cordant. His work spans distributed systems and observability, including autonomous remediation pipelines for large-scale cloud services, with deep experience in Prometheus, OpenTelemetry, and Kubernetes internals.