Why does a load balancer that looks perfectly healthy on a dashboard still take down an application? Ask anyone who has been paged at 3 a.m. and they will tell you the same thing: the load balancer is rarely the thing that broke. It is the thing that faithfully forwarded traffic into something that was already broken, or it stopped forwarding traffic to something that was fine. Either way, the blast radius lands on the load balancer’s reputation. (Load balancing (computing)))
That gap between “the metrics look green” and “users are seeing errors” is where most cloud load balancing problems live. This article walks through the failures that show up again and again in real environments, why they happen, and what actually reduces the pain. (Matching cloud regions to workload requirements)
Health Checks That Lie in Both Directions
Health checks are the single most misunderstood knob in any load balancer configuration. They are also the most common root cause of self-inflicted outages.
A health check is a probe the load balancer sends to each backend on a schedule. Backends that fail enough consecutive checks get pulled out of rotation. That sounds simple, but the failure modes run in two directions.
False negatives happen when the check is too shallow. A plain TCP port check tells you the socket is open. It tells you nothing about whether the application can reach its database, whether a background worker has deadlocked, or whether the process is returning 500s to every real request. A backend that accepts connections but serves nothing useful will happily pass a port check forever.
False positives happen when the check is too strict or too aggressive. Point a health check at an endpoint that does a deep dependency walk, and a brief hiccup in a shared cache can eject every instance at once. Set the interval to five seconds with a threshold of one, and a garbage collection pause becomes a removal event. Now the remaining instances absorb double the traffic, slow down, fail their checks, and get removed too. That is how a small blip becomes a full outage.
What actually works in practice is a dedicated health endpoint that checks only what the instance needs to serve traffic, kept deliberately cheap. Dependency health belongs in monitoring and alerting, not in the rotation decision. It also helps to set the unhealthy threshold higher than the healthy threshold, so instances recover quickly but leave slowly.
Sticky Sessions Quietly Undo Horizontal Scaling
Session affinity, often called sticky sessions, pins each client to one backend. It is the fastest way to make a stateful application work behind a load balancer, and it is one of the most reliable ways to create uneven load later.
The mechanics are unforgiving. If one backend holds sessions for a disproportionate share of active users, it carries more work than its peers regardless of what the algorithm intends. When that backend is replaced during a deployment, every session pinned to it is lost. Users get logged out mid-checkout. Support tickets arrive before the deploy pipeline finishes reporting success.
The practical fix is to externalize session state into a shared store so any backend can serve any request. Where affinity is genuinely required, prefer cookie-based affinity over source-IP affinity. Source-IP affinity breaks badly behind corporate NAT, where thousands of users share one address and all land on the same backend.
Connection Storms and the Failover Stampede
When a backend or an availability zone drops out, every client reconnects at once. This thundering herd effect can overwhelm the capacity that remains, turning a partial failure into a total one.
Two settings matter more than most teams expect. The first is connection draining, also called deregistration delay. It keeps in-flight requests alive on an instance that is being removed, so a scale-in event does not sever active uploads and long-running queries. The second is retry behavior. Clients that retry immediately and without jitter amplify the surge. Exponential backoff with randomization spreads reconnections across time instead of stacking them into a single spike.
There is a subtler version of this problem: pre-warming. Load balancers and autoscaling groups have limits on how fast they can absorb new connections. A marketing launch that triples traffic in ninety seconds can hit those limits before any backend is stressed.
Uneven Distribution Across Availability Zones
Multi-zone architectures look balanced on an architecture diagram and often are not balanced in production. The culprit is usually cross-zone load balancing, a setting that controls whether a load balancer distributes traffic evenly across all registered targets or proportionally across the zones it is enabled in.
Turn cross-zone balancing off and traffic is distributed per zone. If one zone has four healthy targets and another has one, the zone with one gets the same share of traffic as the zone with four. That single target takes four times the load per instance. It degrades first, fails its health checks, and the zone goes dark.
Defaults differ by load balancer type and have changed over time, so teams should verify the current behavior for their specific service rather than assuming. The general rule holds: in a multi-zone deployment, confirm that the distribution you designed is the distribution you are getting.
Idle Timeout Mismatches and Intermittent 502s
Few problems waste as much engineering time as the intermittent 502 or 504 that appears only under light traffic. The usual cause is a mismatch between the load balancer’s idle timeout and the backend’s keep-alive setting.
Here is the pattern. The load balancer holds a connection open to a backend. The backend decides the connection has been idle too long and closes it. The load balancer, unaware, sends a new request down that just-closed connection. The request fails. Under heavy traffic, connections never sit idle long enough to be closed, so the bug hides. Under light traffic, it surfaces constantly.
The fix is to make the backend’s keep-alive timeout longer than the load balancer’s idle timeout, so the load balancer always closes first. It is a one-line configuration change that resolves a class of bugs teams have chased for weeks.
TLS Termination and Certificate Blind Spots
Terminating TLS at the load balancer centralizes certificate management, which is good. It also creates a single point where an expired certificate takes down every service behind it at once.
Certificate expiry remains one of the most preventable causes of outages in cloud environments. The failure is rarely technical. It is organizational: the renewal happened, but the new certificate was installed on the wrong listener, or the automated renewal silently failed months ago, or nobody owned the domain after a team reorganization.
Cipher policy mismatches cause a quieter problem. An old client that cannot negotiate the load balancer’s modern cipher suite gets a handshake failure, and the error surfaces as a vague connectivity complaint rather than a clean application error. Inventorying which clients still connect with legacy protocols is unglamorous work that prevents a specific kind of support escalation.
Source IP Loss and Proxy Protocol Confusion
Layer 7 load balancers terminate the client connection and open a new one to the backend. The backend therefore sees the load balancer’s IP address, not the client’s. Applications that log, rate-limit, geolocate, or block by client IP suddenly behave strangely, and the logs become useless for investigation.
The standard remedies are the X-Forwarded-For header for HTTP traffic and the PROXY protocol for TCP traffic. Both work well and both fail in predictable ways. X-Forwarded-For is trivially spoofable if the application trusts it from any source, so it must only be honored from known proxy addresses. PROXY protocol requires backend support; enable it on a backend that does not understand it and every connection fails immediately.
Algorithm Mismatch: Round Robin Versus Least Connections
Round robin distributes requests in sequence, ignoring how busy each backend actually is. It works beautifully when every request costs the same and every backend performs identically. It works badly when one request is a report generation and the next is a static lookup.
Least connections sends new requests to the backend with the fewest active connections, which handles variable request cost far better. Weighted variants let you account for heterogeneous instance sizes. The mistake is choosing an algorithm once and never revisiting it as the workload changes. A service that started as uniform API calls and grew a heavy export feature may need a different algorithm than the one chosen at launch.
Observability Gaps That Slow Every Investigation
The most expensive load balancing problem is the one nobody can diagnose quickly. Load balancer metrics and application logs frequently live in different tools, with no shared identifier to join them.
Target response time, healthy host counts, and error rates by status code tell you what the load balancer saw. Application logs tell you why. Without correlation, an engineer spends the first twenty minutes of an incident establishing basic facts instead of fixing anything.
Two habits pay off. First, alert on leading indicators such as rising target response time and shrinking healthy host counts instead of relying solely on error rate spikes. Second, make sure the load balancer’s request identifier propagates into application logs so a single failing request can be traced end to end. Teams that get this right consistently resolve incidents in minutes rather than hours.
DNS-Based Balancing and the TTL Trap
DNS-based load balancing looks attractive because it is simple and cheap. It also cannot react faster than client DNS caching allows. A record with a one-hour TTL means a failed endpoint keeps receiving traffic for up to an hour after you remove it. Lowering the TTL helps, but resolvers do not always honor short TTLs, and the propagation is never instant.
DNS balancing belongs in the toolkit for coarse geographic routing and disaster recovery between regions. It is a poor fit for anything that needs fast failover within a region. (9481.org)
A Practical Checklist Before the Next Incident
Most of these problems share a root cause: a configuration that was correct at launch and drifted out of alignment with the workload. A short review catches a surprising number of them.
- Confirm every health check targets a cheap, meaningful endpoint and that thresholds differ for healthy and unhealthy transitions.
- Verify whether session affinity is enabled and whether session state is externalized.
- Check that connection draining is on and that client retries use backoff with jitter.
- Confirm cross-zone balancing behavior matches the intended design.
- Compare the load balancer idle timeout against backend keep-alive settings.
- Inventory certificate expiry dates and who owns renewal.
- Confirm client IP handling is correct for logging and rate limiting.
- Revisit the balancing algorithm against current traffic patterns.
- Verify that load balancer metrics and application logs can be joined.
For teams planning broader changes, it helps to review how workload placement decisions interact with these settings, including when to split workloads across cloud accounts and how matching cloud regions to workload requirements affects latency and failover behavior. Getting the architecture right upstream makes the load balancing configuration far less fragile downstream.
Conclusion
Cloud load balancing problems are rarely exotic. They are health checks that measure the wrong thing, affinity that defeats scaling, timeouts that disagree with each other, and certificates that expire while everyone assumes someone else is watching. None of these require a redesign to fix. They require someone to look at the configuration with fresh eyes and ask whether it still matches how the application actually behaves today.
The teams that handle this well treat load balancer configuration as living infrastructure rather than a one-time setup task. They review it after every significant change in traffic shape, they alert on the early signals, and they keep the investigation path short. That discipline is what separates a load balancer that quietly does its job from one that becomes the headline of a postmortem.
FAQ
What is the most common cause of load balancer outages?
Misconfigured health checks top the list. A check that is too shallow keeps sending traffic to broken backends, while a check that is too strict ejects healthy ones during brief hiccups. Both directions produce outages that look like application failures but originate in the load balancer configuration.
Why do users get logged out during deployments?
Session affinity is usually the reason. When sticky sessions pin a user to a specific backend and that backend is replaced during a rolling deployment, the session is lost. Externalizing session state into a shared store lets any backend serve any request and removes the problem.
What causes intermittent 502 errors under light traffic?
An idle timeout mismatch between the load balancer and the backend. The backend closes an idle connection, the load balancer sends a request down that closed connection, and the request fails. Setting the backend keep-alive timeout longer than the load balancer idle timeout resolves it.
Does cross-zone load balancing matter in a multi-zone setup?
Yes, significantly. With cross-zone balancing disabled, traffic is distributed per zone rather than across all targets, so a zone with fewer healthy instances overloads them. Defaults vary by load balancer type and have changed over time, so verify the current behavior for your service.
How do I preserve the real client IP address?
For HTTP traffic, use the X-Forwarded-For header and configure the application to trust it only from known proxy addresses. For TCP traffic, use the PROXY protocol, which requires backend support. Trusting forwarded headers from any source allows trivial spoofing.
Can DNS load balancing replace a load balancer?
Not for fast failover. DNS-based balancing is limited by client caching, so a failed endpoint can keep receiving traffic for the full TTL after removal. It works well for coarse geographic routing and cross-region disaster recovery, but not for rapid failover within a region.
