At 2am, can your dashboards tell you what changed, who is affected, and what caused it? Observability has advanced substantially in recent years. Metrics, logs, and traces are standard, structured logging is widely adopted, and distributed tracing covers request paths across microservices. Many organizations have more data than engineers can navigate.
Better tooling has not guaranteed a better on-call experience. Diagnosis still takes 15 to 30 minutes because the needed data exists but is not presented to answer the three questions that matter at 3am: is the situation worsening, how broad is the impact, and where is the probable cause?
This is not a guide to comprehensive observability architecture. It focuses on what an on-call engineer needs during an active incident and how to arrange the stack for that narrow but critical purpose.
Three Questions Before the Data
Incident response generally follows this sequence: establish scope and severity, develop a working cause hypothesis, test it, then resolve or escalate. Each phase calls for different observability data.
For scope and severity, start with user-impact measures: error rate, latency percentiles, and availability for each user-visible surface. Internal system metrics can wait. First establish what users experience, not the reason behind it.
When forming a hypothesis, connect user-impact measures to internal system metrics. Identify where internal values changed and when that happened relative to the user-facing symptom. Correlation on the timeline is the diagnostic lever.
Testing a hypothesis requires narrowing the query: this service, this time window, this error code. High-cardinality metrics and structured logs with arbitrary filtering are most useful at this stage.
Designing the stack around these phases creates three layers: impact first, correlation second, and detail third. Engineers should move between them quickly without knowing beforehand exactly what they need to find.
The On-Call Dashboard Problem
Most monitoring dashboards serve capacity planning, trend analysis, and normal system behavior, not active incident response. Those are valid uses, but they often yield 40+ panels containing every metric the team tracks, arranged by service or component instead of diagnostic workflow.
During an incident, an on-call engineer must scan 40 panels to find anomalies. That is manageable in normal conditions, but sleep deprivation and time pressure make the process slow and error-prone.
The fix is a separate incident dashboard built around the three phases above. Put an impact summary first, with overall error rate, latency, and availability in one row. Next, show correlation: error rate and latency for each service side by side, so one service's deviation is visible against the others. Then link to each service's detail dashboard. This dashboard is for triage. It points the engineer to the right destination instead of demanding extended investigation there.
Logs: Structured Data That Cannot Be Queried
Structured logging is nearly universal. Its advantages are real: machine-readable lines, arbitrary field filtering, and aggregation across services. During incidents, though, it works only when diagnostic fields are present and named consistently everywhere.
Many observability implementations fail here. Service A records a user identifier as user_id, Service B as userId, and Service C as uid. Service D omits it. If an incident requires tracing a request through four services, inconsistent names make correlation manual and slow.
A platform team can make one of its most productive observability investments by standardizing a small set of correlation fields across every service. Request ID, user identifier, or session identifier for unauthenticated sessions, service name, and error code should use the same names and formats everywhere. Then one query can correlate logs across services, instead of separate queries and manual matching.
Distributed Tracing, Limits and Uses
Distributed tracing shows the complete request path across services, with timing for every hop. It is genuinely invaluable when diagnosing latency in systems with many dependencies. If a request took 1200ms and 1150ms occurred in a database query, the diagnosis is unambiguous.
Tracing is less useful in two common situations. First, load may cause sampling to discard a significant share of traces. A 10% sampling rate can represent normal traffic, yet during an incident with elevated errors, an error-unaware strategy may discard errors disproportionately. Sample errors at a higher rate than successful requests.
The second case is an incident involving a component without tracing instrumentation. A trace is complete only when every service on the request path emits trace data. One untraced service leaves a gap and an incomplete timeline. During investigation, that missing information is a serious problem because it often sits exactly where the failure is.
The Time Correlation Invariant
One consistently neglected observability property is also unusually valuable during incident response: close time alignment across every data source.
A 5-minute metrics window, second-level log timestamps, and millisecond-spanning traces have different resolutions, so correlating them requires adjustment. When identifying which service changed first in a cascading failure, 5-minute metric granularity may not resolve the sequence.
Ask this: for your most critical services, what is the metric time resolution, and can it establish causal order in a cascading failure? For most platform services, 1-minute resolution is the minimum for meaningful incident timeline analysis. Use 30-second resolution where fast-moving failures are common.
Observability Cannot Fix Architectural Problems
We should be direct: better observability does not repair systems that are difficult to diagnose because of their design. Monolithic deployments where many services change together obscure what changed before an incident. Tightly coupled services let one failure cascade through others, obscuring the root service. Missing error boundaries make diagnostic scope difficult to contain.
Good observability speeds diagnosis in a diagnosable system. It cannot make an inherently opaque system clear. If post-mortems repeatedly show that diagnosis took a long time because the system was difficult to reason about, not because data was absent, invest in architectural clarity instead of more observability tooling.
Fast incident response requires both useful data and a system whose data can be interpreted. Neither side is sufficient alone.