Availability percentage is widely tracked but predicts very little. Most reliability metrics are lagging indicators: error rate rises, latency exceeds the SLO threshold, or availability falls. These signals confirm real problems, but only after they occur. When they fire, users are often already seeing degraded service.
Leading indicators track conditions that come before incidents, creating time to intervene before users are affected. They are harder to define, vary by service, and need comparison with historical incident data before they are trustworthy. Standard dashboards rarely contain them. You must create them for the system you operate.
This post examines leading-indicator categories that have shown predictive value across the services we studied. It also explains how to test whether a metric predicts incidents or merely correlates with another predictive signal.
Resource Saturation Trajectories, Not Thresholds
Fixed resource limits, such as CPU above 80% or memory above 70%, lag because performance may already suffer when those thresholds are reached. A better signal is the change in resource consumption relative to current utilization.
A service at 40% CPU, rising 5 percentage points per hour on its current traffic, will reach saturation in 12 hours. That provides a defined action window. Another service at 75% CPU with no upward trend is not in immediate danger, despite its greater current utilization.
Track utilization velocity and estimated time-to-saturation under current traffic growth, not utilization alone. This takes more monitoring than a single threshold alert, but it can provide hours of warning instead of seconds.
Retry Storms as Early Error Signals
Retry storms frequently trigger cascading failures in microservice architectures. A downstream dependency slows, clients retry, and the extra traffic slows it further. More clients then retry until the dependency fails completely and pulls its dependents down with it.
Watch for rising retry rate rather than waiting for error rate to rise. Elevated client errors may arrive after the retry storm is already underway. A rising retry rate alongside stable success rates is earlier evidence: the system is still succeeding, but at greater effort.
For each client, track total requests divided by successful requests, grouped by downstream service. If the usual ratio of 1.02 to 1.05 (minor retry overhead is normal) rises to 1.2 or 1.3, investigate before it turns into a paging event.
Connection Pool Utilization Patterns
Database connection pool exhaustion commonly causes incidents, and monitoring data usually contains a clear early signal. Pools have fixed sizes. As utilization nears 80 to 90% of capacity, exhaustion becomes substantially more likely during a traffic spike or slow-query event.
The useful signal is not current pool utilization alone, but the highest utilization seen over the last N minutes compared with pool capacity. A pool that often reaches 95% under normal load needs only one slow query or traffic spike to exhaust. A configuration that worked at lower traffic can become risky as the service grows.
Monitoring dashboards often hide connection pool metrics by default. They generally require explicit instrumentation. When the metric is missing from the standard dashboard, that usually indicates an observability gap, not that the underlying problem is absent.
When Async Queues Start Falling Behind
Asynchronous services have a natural early signal: queue-depth movement compared with processing rate. When enqueueing outpaces processing, the queue grows. Growth is not itself an incident, but it reliably points toward higher future latency as wait time increases, and possible failures if backing storage fills or messages expire.
Measure queue-depth velocity: the rate at which the queue grows or contracts. A queue holding 10,000 messages but declining by 1,000 messages per minute is healthy. The same depth, increasing by 500 messages per minute under normal load, indicates a capacity issue that will become an incident if left unresolved.
In Kafka, Kinesis, and Pub/Sub, consumer lag is this metric's standard form. Track its velocity, not just the absolute value.
Testing a Metric's Predictive Value
To validate a leading indicator, select historical incidents from one category, such as database connection pool exhaustion incidents over the past 12 months. Examine the candidate metric during the 60 to 120 minutes before each incident and determine whether it displayed an anomalous pattern.
A useful leading indicator appears in at least 70% of incidents in its category, providing good recall, and produces false positives less than 20% of the time when no incident occurs, providing acceptable precision. Below those thresholds, refine the metric before adding it to the alerting layer.
Historical incident data is necessary for this validation. Real incidents provide the test cases. Maintaining an incident corpus with metric data from each incident window therefore supports more than post-mortems. It supplies training data for identifying your system's predictive signals.
The Limits of Leading Indicators
Leading indicators do not replace lagging indicators. They must be calibrated to your system and traffic patterns, and a useful signal for one service may be noise in another. Their greatest value comes in familiar systems, recurring incident categories, and situations where historical data has validated them.
They cannot help with novel or genuinely surprising failures. A metric defined in advance cannot predict a failure mode no one anticipated. Anomaly detection addresses that gap, but brings its own tradeoffs in false positive rate and detection latency.
Their practical value is greatest for recurring incident categories. If connection pool exhaustion occurs 10 times per year and a dependable indicator fires 60 minutes before each event, those 10 responses become 10 proactive interventions. User impact falls significantly without changing response team capacity.