A pod can return HTTP 200 to its liveness probe for hours while its p99 request latency climbs from 180 ms to 4 seconds, its dependency calls time out, and users receive a steady stream of 5xx responses. Kubernetes still sees a healthy container.
That is not a defect in the kubelet or in health probes. It is a mismatch between the question a probe answers and the question an operator needs answered during an incident.
Liveness and readiness probes are useful control signals. They help Kubernetes decide whether to restart a container or include a pod in a Service's endpoints. They do not describe the full health of a production system. They do not explain why traffic is failing, how widely the failure is distributed, or whether restarting a pod will improve anything.
A probe reports a narrow, binary contract
A liveness probe answers a question close to: "Should this container continue running?" A failed liveness probe can cause the kubelet to restart the container after the configured failureThreshold is reached.
A readiness probe answers a different question: "Should this pod receive traffic through its Service?" When readiness fails, Kubernetes can remove the pod from the set of ready endpoints. The container remains running, but normal Service routing should stop sending it new requests.
Both signals reduce a complex state to success or failure. An HTTP probe commonly treats a response in the 200 to 399 range as successful. A TCP probe needs a connection. An exec probe needs the command to exit successfully. A gRPC probe needs the expected health response. None of these outcomes captures the shape of degradation.
There is no standard probe result for "the endpoint is succeeding, but p99 latency is above the error budget," or "database calls are timing out for 7 percent of requests," or "only tenants routed to one shard are failing." Those are observability questions.
Where the binary model loses important context
The first gap is partial failure. A service can serve cached responses successfully while fresh reads fail. It can return 200 for a lightweight health endpoint while the main request path is blocked on a saturated connection pool. It can pass a readiness check from one node while requests from another availability zone encounter packet loss.
The second gap is dependency behavior. A probe that checks only process health says little about an overloaded database, a slow queue, an unavailable object store, or a misconfigured downstream API. Adding every dependency to a readiness endpoint may expose some of that state, but it also creates a new tradeoff: a transient dependency failure can remove every pod from service at once, amplifying the outage rather than containing it.
The third gap is time. Probe configuration has sampling behavior. With a 10-second periodSeconds, a short failure may never be observed. With a low failureThreshold and a short timeoutSeconds, normal startup or temporary load spikes can trigger avoidable restarts. A startupProbe can protect slow-starting applications by delaying liveness and readiness checks, but it does not measure whether the application remains healthy after startup.
The fourth gap is causality. A failed readiness probe identifies a pod that should not receive traffic. It does not identify whether the cause is a bad deploy, exhausted CPU, a thread pool deadlock, a DNS problem, or a dependency regression. The kubelet records probe results and takes the configured action. It is not an incident investigator.
A realistic failure pattern
Consider a platform team at a mid-size cloud services company running 40 to 50 services on EKS. One API deployment has 12 replicas and a readiness endpoint that checks process state and the ability to acquire a database connection. Its liveness endpoint checks only that the HTTP server can respond.
A schema change causes a query plan to degrade for a high-volume customer segment. The database remains reachable, so readiness continues to pass. The application process is not deadlocked, so liveness continues to pass. All 12 pods remain ready and running.
From Kubernetes' perspective, the deployment is healthy. From the user's perspective, requests for that segment now take 8 to 12 seconds, exceed the ingress timeout, and produce elevated 504 responses. Prometheus shows rising request duration, a higher error ratio, and database wait time increasing. The probe dashboard remains green.
Restarting the pods is unlikely to fix the query plan. Removing them from the Service is also unlikely to help if every replica shares the same database behavior. The useful operational response is to connect symptoms across layers: route or tenant dimensions, application latency, database wait metrics, deployment history, and the error budget impact.
That distinction matters during automation. A restart is a reasonable response to a process that has stopped making progress. It is a poor response to a shared dependency that is slow but reachable. Treating every unhealthy user experience as a pod-health problem can turn a degraded service into a restart loop.
What observability adds
Observability starts with signals that preserve more information than a Boolean probe result.
- Request rate: Are requests arriving as expected, and is the affected traffic concentrated in one route, tenant, region, or status class?
- Error rate: Are failures 4xx responses, 5xx responses, timeouts, cancellations, or retries that never reach the application?
- Duration: What are p50, p95, and p99 latencies, and which route or dependency accounts for the tail?
- Resource saturation: Are CPU throttling, memory pressure, file descriptors, connection pools, worker queues, or ephemeral storage approaching a limit?
- Dependency health: Are database wait times, queue age, DNS resolution, downstream latency, or retry counts changing at the same time?
This is the practical value of the RED method for services: request rate, errors, and duration expose gradual degradation that probes flatten into "pass." Kubernetes metrics add scheduling and container context, while logs and traces can connect a failed request to a specific dependency or code path.
Probe results still belong in this picture. A readiness failure can explain why traffic shifted. A liveness failure can explain a restart. Container restart counts, pod events, rollout state, and probe latency are useful evidence. They are simply not sufficient evidence on their own.
Design probes for traffic management, not diagnosis
Probe endpoints should have an explicit operational purpose. Keep liveness checks narrow enough that a temporary dependency outage does not cause a fleet-wide restart. Make readiness reflect whether the pod can safely accept the traffic it is likely to receive, while considering whether dependency checks could remove capacity too aggressively.
Use startupProbe for applications with predictable but lengthy initialization. Set initialDelaySeconds, periodSeconds, timeoutSeconds, and thresholds from observed startup behavior rather than copied defaults. A probe that times out because the application needs 900 ms under normal load is not measuring the same condition as a probe that times out because the process is wedged.
For diagnosis, pair those controls with service-level indicators and alerts. An alert on readiness failures alone tells the on-call engineer that Kubernetes is withdrawing capacity. An alert that combines elevated 5xx rate, p99 latency, pod restarts, and a recent rollout provides a much narrower set of plausible causes.
We are not saying health probes are unimportant, or that every readiness endpoint should become a miniature observability system. We are saying that probes are traffic and process-management mechanisms, while observability is what lets an engineer understand system behavior and choose a safe response.
The nuance: more checks can create new failure modes
It is tempting to fix probe blindness by putting more checks into the readiness endpoint. That can work for a carefully chosen dependency, but it is not free. If readiness depends synchronously on three downstream systems, a short outage in any one of them can make every pod unready. If the endpoint performs an expensive database query, the health check itself consumes capacity during the incident.
There is also a difference between removing a pod from new traffic and stopping work already in progress. Readiness changes do not instantly cancel existing requests. Connection draining, load balancer behavior, termination grace periods, retries, and client-side connection reuse all affect how quickly traffic actually moves.
Probe failures can therefore be noisy without being useless. They should be interpreted with rollout events, application metrics, node conditions, and dependency signals. Sometimes the correct action is to restart one stuck process. Sometimes it is to pause a rollout, reduce retry pressure, shed a class of traffic, or wait for a dependency to recover. The probe supplies one piece of that decision.
From green probes to better incident decisions
For a platform team, the practical test is not whether every pod is green. It is whether an engineer can answer four questions quickly: what user behavior is failing, when did it begin, which component introduced the change, and what action is reversible?
That is the operating gap Cordant is built around. Cordant watches production systems, diagnoses incidents as they start, and proposes or applies a fix, waking an on-call engineer when a human decision is genuinely required. Probe state can be part of that evidence, but it should be correlated with the signals that show actual service impact.
A green readiness dashboard during rising p99 latency is not contradictory. It is a reminder that the dashboard is reporting endpoint eligibility, not the complete health of the service. In a real incident, the useful boundary is the one between "Kubernetes has taken an action" and "we understand why users are affected."