The First Lie: “Perfect Balance”

Load balancing is usually explained like this: “We evenly distribute traffic across servers.”

That idea is comforting. It is also mostly fictional. In real systems:

• requests have different costs
• servers have different speeds
• traffic arrives in bursts

Visualizing the Problem

Let’s start simple: one load balancer sending requests to a single server. Requests are green dots. They shrink as the server processes them.

For many websites, this works fine. But what happens when traffic increases?

Round Robin

The simplest solution: add more servers and send requests to each one in turn. This is called round robin.

Beautiful. Each server gets the same number of requests, queues stay balanced. This is why round robin is the default in nginx.

When Round Robin Breaks

In the real world, requests aren’t equally expensive. An image resize takes longer than a health check. Let’s add some variance:

Round robin balances requests, not work. A server that gets three expensive requests in a row will fall behind.

It gets worse with heterogeneous servers:

The fix? Weighted round robin. If a server is 3x faster, give it 3x the traffic:

Request Queues

You might have noticed the servers have queues (the vertical bars above them). Without queues, any request that arrives while the server is busy gets dropped immediately.

Queues let us absorb bursts, but they’re a trade-off: we drop fewer requests, but at the cost of higher latency. A request sitting in a queue isn’t being processed—it’s waiting.

Least Connections

What if instead of blindly rotating, we sent each request to the server with the shortest queue?

Notice how the queues stay much more balanced? Least connections cuts through uncertainty by tracking actual work in progress.

The Power of Two Choices

What if instead of checking every server (expensive) or picking one at random (chaotic), we did something in between?

Idea: Pick 2 random servers. Send the request to whichever has less load.

Algorithm Comparison

Let’s put them head to head. Same high-variance scenario, different algorithms. Watch how they handle the chaos:

Notice the difference in dropped requests and queue balance. Least connections adapts to the actual work being done.

Real-Time Benchmark

Watch all four algorithms compete in real-time. The simulation runs identical workloads through each strategy simultaneously, tracking latency, drop rate, and throughput.

Performance Metrics

The simulations above show behavior in real-time, but let’s look at the aggregate data. These charts show what happens over thousands of requests:

Hash-Based Load Balancing

Sometimes you need requests from the same user to hit the same server. Session affinity, caching, stateful connections.

Idea: Hash something stable (user ID, session ID, IP) and use it to pick a server deterministically.

The Celebrity Problem

If you hash by user ID and @elonmusk tweets, the server owning that hash bucket melts. Then health checks fail, traffic shifts, the celebrity moves to the next node, and that node melts too.

Choosing an Algorithm

• Uniform traffic, identical servers → Round Robin
• Variable request costs → Least Connections
• Large server pools → Power of Two Choices
• Session affinity needed → Consistent Hashing

Playground

Now it’s your turn. Experiment with different algorithms and parameters. Try to break things.

Conclusion

The best load balancing algorithm is often “buy more servers.” The second best is “fix your database queries.”

If you are desperately tuning load balancing algorithms to survive, you probably don’t have a routing problem. You have a capacity problem.

Load balancers distribute traffic. They do not create headroom.