What happens when one server in your fleet quietly starts returning errors, but the load balancer keeps sending it a third of your traffic? That single scenario explains why so many cloud outages trace back not to the database or the app code, but to a load balancer that was configured once and never tuned. The good news: most of the failure modes are predictable, and a handful of practices prevent them.
This guide walks through the decisions that matter most, from the Layer 4 versus Layer 7 choice to health check tuning, connection draining, and observability. It is written for teams running real workloads on AWS, Google Cloud, Azure, or a mix, where “it works in staging” is not good enough.
Start With the Layer 4 Versus Layer 7 Decision
Before touching a single setting, decide what kind of balancer you need. This is the first fork in the road, and getting it wrong means fighting the tool for the rest of the project.
Layer 4 (transport) balancers route TCP and UDP traffic without inspecting the payload. They are fast, cheap, and preserve connections. Layer 7 (application) balancers understand HTTP and HTTPS, which supports path-based routing, host-based routing, TLS termination, header manipulation, and content-aware rules. AWS documents this split across its Elastic Load Balancing family, and most cloud providers use the same division.
A practical rule of thumb: if you need to route `/api` to one service and `/static` to another, or terminate certificates at the edge, you want Layer 7. If you are balancing raw database connections, game servers, or custom protocols, Layer 4 is the right tool.
The most common mistake here is reaching for Layer 7 by default because it feels more “modern,” then paying for it in latency and complexity on workloads that never needed HTTP awareness. The second most common mistake is the reverse: trying to do path routing on a Layer 4 balancer by running a reverse proxy behind it, which adds a hop and a failure domain you did not plan for.
Tune Health Checks So One Blip Does Not Take Down a Backend
Health checks are the nervous system of your load balancer. They decide which backends receive traffic, and they are almost always misconfigured in the same three ways.
First, teams use the default probe interval and thresholds without thinking. If the interval is 10 seconds and the unhealthy threshold is 2, a backend gets ejected after roughly 20 seconds of failures. That may be fine, or it may eject a healthy instance during a brief garbage collection pause. Set the unhealthy threshold high enough that a single failed probe does not remove a working server.
Second, teams point health checks at the homepage or a generic endpoint that does not reflect real dependency health. A better pattern is a dedicated health endpoint that checks the database connection pool, cache availability, and any critical downstream service, then returns a status code accordingly. Google Cloud’s documentation on health checks describes how probe types, intervals, timeouts, and thresholds interact, and it is worth reading before you set values.
Third, teams forget the healthy threshold. After a backend recovers, it needs to pass a number of consecutive checks before it rejoins. Set that too low and a flapping instance will bounce in and out of rotation, creating a thundering herd effect on every recovery.
A Health Check Tuning Checklist
- Use a dedicated `/health` endpoint that tests real dependencies rather than process liveness alone.
- Set the unhealthy threshold to at least 3 so a single timeout does not eject a healthy node.
- Set the healthy threshold to at least 2 so recovered nodes prove stability before taking traffic.
- Keep the timeout shorter than the interval, or checks will stack up.
- Test the failure path deliberately by killing a backend and watching how long removal takes.
Get Cross-Zone Distribution Right
Load balancers distribute traffic across availability zones, and the default behavior here has changed over time on major clouds. If cross-zone load balancing) is off, each zone’s balancer nodes only send traffic to targets in their own zone. That sounds fine until one zone has three healthy targets and another has one, and the imbalance creates hot spots.
Enabling cross-zone distribution flattens the load across all registered targets regardless of zone. The trade-off is cross-zone data transfer cost, which is exactly the kind of hidden line item that shows up on the bill three months later. Before you flip the switch, model the egress. A clearer way to track cloud data transfer costs walks through how to attribute that spend before it surprises you.
The decision also depends on how you have split your environments. Teams that run separate accounts per environment or per business unit face different balancing trade-offs than teams running one shared account, and the reasoning behind when to split workloads across cloud accounts is worth understanding before you design the topology.
Connection Draining Is Not Optional
Here is a scenario that produces mysterious 502 errors: an autoscaling group scales in, a backend is terminated, and requests that were mid-flight are dropped. The fix is connection draining, sometimes called deregistration delay. When a backend is removed, the balancer stops sending new requests but lets in-flight ones finish within a timeout window.
The default is often 300 seconds, which is generous for most APIs and far too long for a service that needs to scale down fast. Set the delay slightly above your application’s longest expected request time. If your p99 request takes 8 seconds, a 30-second drain is plenty. If you run long-lived streaming connections, you need a different strategy entirely, often a graceful shutdown signal in the application itself.
The practical tip: measure your actual longest request, do not guess. Pull the p99 and p99.9 latency from your access logs, add a safety margin, and set the drain window from that number. AWS documents the deregistration delay and target health settings in its target group documentation, which is the right place to confirm current defaults for your balancer type.
Decide TLS Termination and Re-Encryption Separately
Teams often treat “TLS at the load balancer” as one decision. It is really two. The first is whether to terminate TLS at the balancer at all. The second is whether to re-encrypt traffic from the balancer to the backends.
Terminating at the balancer simplifies certificate management enormously, especially when you use managed certificates that renew automatically. The cost is that traffic is decrypted at the edge, which matters for compliance regimes that require end-to-end encryption. Re-encrypting to backends restores that end-to-end property at the cost of extra compute and slightly higher latency. AWS Certificate Manager documentation covers how managed certificates integrate with load balancers if you want the details.
A balanced approach many teams land on: terminate at the balancer for public traffic, re-encrypt to backends for anything touching sensitive data, and skip re-encryption for internal services already inside a private network with its own controls.
Use the Balancer’s Own Metrics to Drive Autoscaling
CPU utilization is a poor proxy for demand on I/O-bound services. A service waiting on a database can sit at 20% CPU while request queues grow. Better scaling signals come from the load balancer itself: request count per target, active connection count, and target response time.
Both major clouds support scaling policies based on these metrics. The pattern that works: scale out on request count per target, scale in on a slower, more conservative policy so you do not thrash. Pair that with a minimum capacity floor that can absorb a sudden spike while new instances boot, because instance startup time is the real constraint on how fast autoscaling can respond.
If you are planning a migration and want to get these baselines right before cutover, it helps to know what to inventory before a cloud migration, since load balancer configuration and traffic patterns are easy to overlook in the planning phase.
Treat Sticky Sessions as a Trade-Off, Not a Default
Session affinity, or sticky sessions, routes a user’s requests to the same backend using a cookie or source IP. It solves real problems for stateful applications, but it concentrates load and complicates failover. When a sticky backend dies, every user pinned to it loses their session.
The better long-term pattern is stateless application servers with session state in an external store like Redis or a database. Sticky sessions then become unnecessary, and any backend can serve any request. If you must use them, set the stickiness duration deliberately and understand that it interacts with connection draining: a draining backend with active sticky sessions needs to keep serving until those sessions expire or migrate.
Log and Measure at the Balancer
The load balancer sees every request, which makes it the best place to observe. Access logs, target health metrics, and latency percentiles (p50, p95, p99) are the primary signals for tuning. AWS exposes these through Application Load Balancer access logs, and other clouds offer equivalents.
Three metrics deserve a dashboard: the rate of 5xx responses by target, the number of unhealthy targets over time, and the gap between p50 and p99 latency. A widening gap usually means a few slow backends are dragging down the pool, which is a signal to investigate before users notice.
For teams building out a broader observability practice, the cloud infrastructure and architecture resources at 9481.org cover how these signals fit into a larger operating model, and the enterprise AI and cloud security guidance explains how to keep that telemetry protected. (9481.org)
Design for Failure, Then Test It
Every practice above assumes something will break. The teams that handle incidents well are the ones that rehearsed the failure before it happened. That means deliberately killing backends during business hours, watching how quickly the balancer removes them, and confirming that traffic shifts cleanly.
Run a game day once a quarter. Kill a backend. Drain a zone. Watch the dashboards. The first run always surfaces something: a health check that was too aggressive, a drain window that was too short, an autoscaling policy that never triggered. Fix those in a controlled setting, not at 2 a.m.
It also helps to understand how regions and zones factor into your design, since balancing across zones is only as good as the regional topology underneath it. Matching cloud regions to workload requirements covers how latency and compliance shape those choices, and knowing how to read a cloud region announcement helps you plan capacity before a new region goes live.
Conclusion
Cloud load balancing rewards deliberate design and punishes defaults left untouched. The practices that matter most are unglamorous: choose the right layer, tune health checks so they reflect real health, enable cross-zone distribution with the cost modeled, set connection draining from measured latency, and watch the balancer’s own metrics rather than guessing.
None of these require exotic tooling. They require someone to sit down with the configuration, understand what each setting does, and test it under controlled failure. Teams that do this spend far less time firefighting and far more time shipping.
Frequently Asked Questions
What is the difference between a Layer 4 and Layer 7 load balancer?
A Layer 4 balancer routes TCP and UDP traffic without inspecting content, making it fast and cheap but blind to HTTP details. A Layer 7 balancer understands HTTP and HTTPS, enabling path routing, host routing, TLS termination, and header-based rules. Choose Layer 4 for raw protocols and Layer 7 when you need application-aware routing.
How many health check failures should trigger removal of a backend?
At least three consecutive failures is a reasonable starting point, so a single timeout or brief pause does not eject a healthy server. Pair that with a healthy threshold of two or more so recovered backends prove stability before rejoining. Always test the failure path to confirm the timing works for your workload.
What is connection draining and why does it matter?
Connection draining, also called deregistration delay, lets in-flight requests finish after a backend is removed from rotation while new requests go elsewhere. Without it, scaling events and deployments drop active requests, producing 502 errors. Set the window slightly above your measured p99 request time.
Should I enable sticky sessions on my load balancer?
Only if your application keeps session state locally and cannot easily externalize it. Sticky sessions concentrate load and complicate failover because users pinned to a failed backend lose their session. Stateless servers with an external session store are generally the better architecture.
How should I set up autoscaling for a load-balanced service?
Scale on load balancer metrics such as request count per target or active connections rather than CPU alone, especially for I/O-bound services. Use a fast scale-out policy and a slower scale-in policy, and keep a minimum capacity floor that covers instance startup time during sudden spikes.
Why is cross-zone load balancing important?
Without cross-zone distribution, each zone’s balancer nodes only send traffic to targets in their own zone, which creates hot spots when zones have uneven capacity. Enabling it flattens load across all targets, though it introduces cross-zone data transfer costs that should be modeled before you turn it on.
