Load Balancing Strategies

Distribute traffic without bottlenecks

Scaling your website or application efficiently is important for handling increasing traffic and keeping the user experience smooth. One key aspect of scaling is load balancing, which involves distributing incoming network traffic across multiple servers to prevent any single server from becoming overwhelmed. This ensures reliability and high availability.

In this blog post, I'll explore various load balancing strategies, looking at how they work and at the pros and cons of each. Whether you run a website or are just curious about how the web works, understanding these strategies will help you see how websites serve millions of users.

Strategies

There are several load balancing strategies that can be employed to distribute incoming requests across multiple servers. These strategies can be broadly categorized into three groups: Static Load Balancing Methods, Dynamic Load Balancing Methods, and Client-Affinity Load Balancing Methods.

We may also consider combining multiple strategies to create a more robust and efficient load balancing solution. Let's explore each of these strategies in more detail.

Static Strategies

Static strategies follow a fixed rule and do not look at the current state of the servers.

Random Strategy (Not Recommended)

Requests are distributed randomly among the available servers, requiring minimal configuration. This is a naive approach that doesn't consider server load or capacity.

graph LR
  R1[Req#1] --> S{Random}
  R2[Req#2] --> S
  R3[Req#3] --> S
  S -->|Req#1| A[Server 1]
  S -->|Req#2| B[Server 2]
  S -->|Req#3| B[Server 2]
  S -->|NONE| C[Server 3]

We should avoid this strategy for production environments as it can lead to uneven server loads and potential overloading of servers. I am just mentioning it here for the sake of completeness.

Pros	Cons
Simple to implement	Does not account for server load or capacity
No need for complex configuration	Potential for overloading servers

Choose this strategy if:

• You have a small number of servers.
• You are looking for a quick and easy solution.
• You are not concerned about server load balancing!

Round Robin Strategy

The Round Robin strategy distributes incoming requests equally across all servers in a rotating fashion. It's simple and doesn't require tracking the current load of servers. Requests are assigned to servers in a cyclical manner, ensuring an even distribution of traffic.

graph LR
  R1[Req#1] --> S{Round Robin}
  R2[Req#2] --> S
  R3[Req#3] --> S
  R4[Req#4] --> S
  R5[Req#5] --> S
  S -->|Req#1| A[Server 1]
  S -->|Req#2| B[Server 2]
  S -->|Req#3| C[Server 3]
  S -->|Req#4| A[Server 1]
  S -->|Req#5| B[Server 2]

An improved variation of this strategy is the Weighted Round Robin, which assigns a weight to each server based on its capacity. Requests are distributed according to these weights, allowing for more efficient resource utilization, given that we know the capacity of each server.

graph LR
  R1[Req#1] --> S{RR Weighted}
  R2[Req#2] --> S
  R3[Req#3] --> S
  R4[Req#4] --> S
  R5[Req#5] --> S
  S -->|Req#1| A[Server 1 - 3x]
  S -->|Req#2| B[Server 2 - 1x]
  S -->|Req#3| C[Server 3 - 1x]
  S -->|Req#4| A[Server 1 - 3x]
  S -->|Req#5| A[Server 1 - 3x]

A more complex version of this strategy is the Sticky Round Robin, which ensures that requests from the same client are directed to the same server. This is useful for maintaining session persistence.

graph LR
  R1[User#1 Req#1] --> S{RR Sticky}
  R2[User#1 Req#2] --> S
  R3[User#2 Req#3] --> S
  R4[User#2 Req#4] --> S
  R5[User#1 Req#5] --> S
  S -->|Req#1| A[Server 1]
  S -->|Req#2| A[Server 1]
  S -->|Req#3| B[Server 2]
  S -->|Req#4| B[Server 2]
  S -->|Req#5| A[Server 1]

Pros	Cons
Easy to implement and manage	Plain round robin ignores load and capacity
Ensures equal distribution of requests	Potentially overloading weaker servers

Choose this strategy if:

• You have a small number of servers with similar capacities.
• You want a simple and straightforward load balancing solution.

Dynamic Strategies

Dynamic strategies look at the current state of the servers before they route a request.

Least Load Strategy

This strategy routes traffic to the server with the lowest load. This could be based on CPU usage, memory usage or number of connections. The goal is to ensure that requests are directed to the server that can handle them most efficiently.

graph LR
  R1[Req#1] --> S{Dynamic by CPU}
  R2[Req#2] --> S
  R3[Req#3] --> S
  R4[Req#4] --> S
  R5[Req#5] --> S
  S -->|Req#1| A[Server 1 - 20%]
  S -->|Req#2| B[Server 2 - 80%]
  S -->|Req#3| C[Server 3 - 50%]
  S -->|Req#4| A[Server 1 - 20%]
  S -->|Req#5| C[Server 3 - 50%]

If we are monitoring the server's active connections, we can direct new requests to the server with the fewest. This is the Least Connections method, assuming that less busy servers can handle new requests more efficiently. This method works well when requests vary in complexity and duration. Servers that have fewer connections are prioritized to handle new requests as they are likely to be less loaded.

A less utilized server is probably handling connections faster because its resources are not being used to the maximum or it has more resources available to handle new connections.

Additionally, we may take into consideration the request response time. If a server is responding faster than others, it might be a good idea to direct more traffic to it. This is the Least Response Time method.

Pros	Cons
Adapts to changing server loads	Requires real-time monitoring of connections
Prevents overloading of servers	Increased complexity

Choose this strategy if:

• Your application has varying request loads.
• You want to optimize server performance based on real-time metrics.
• You have the resources to monitor and manage server loads.

Queue-based Strategy

Incoming requests are placed in a queue and then distributed to servers as they become available. This strategy ensures that no server is overwhelmed by distributing work evenly. It can be combined with other strategies for optimal performance.

graph LR
  R1[Req#1] --> Q{Queue}
  R2[Req#2] --> Q
  R3[Req#3] --> Q
  Q -->|Req#1| A[Server 1]
  Q -->|Req#2| B[Server 2]
  Q -->|Req#3| C[Server 3]
  C -->|Failed| Q
  Q -->|Req#3| B[Server 2]

When a server fails to process a request, the request goes back to the queue and another server picks it up for a retry.

Using a queue-based strategy may introduce latency as requests wait in the queue. Additionally, it requires additional systems to manage the queue, adding complexity to the infrastructure. However, it gives the ability to prioritize requests and redistribute them if, for example, a server fails.

Pros	Cons
Ensures no server is overwhelmed	May introduce latency
Can be combined with others	Additional systems to manage

Choose this strategy if:

• You want to ensure that no server is overwhelmed.
• You need to prioritize requests.
• You are willing to manage the additional complexity.

Client-Affinity Strategies

Requests are distributed based on client-specific attributes, such as IP address, URL, cookies, or user-agent. This ensures that a client's requests are consistently directed to the same server, maintaining session persistence and cache efficiency.

graph LR
  R1[Req#1] --> S{IP Hashing}
  R2[Req#2] --> S
  R3[Req#3] --> S
  R4[Req#4] --> S
  R5[Req#5] --> S
  S -->|Req#1-Hash#1| A[Server 1]
  S -->|Req#2-Hash#2| B[Server 2]
  S -->|Req#3-Hash#1| A[Server 1]
  S -->|Req#4-Hash#1| A[Server 1]
  S -->|Req#5-Hash#2| B[Server 2]

In some cases, it may be beneficial to route requests based on the client's geographic location, optimizing latency and content delivery. This strategy is known as Geographic Load Balancing.

Pros	Cons
Guarantees client-server affinity	May not distribute load evenly
Optimizes cache usage	Requires additional client tracking

Choose this strategy if:

• You need to maintain session persistence.
• You want to optimize cache usage.
• You are willing to manage client-specific attributes.

Server Health Management

In addition to load balancing strategies, it's essential to monitor server health and performance. Health Check Monitoring involves regularly checking the status of servers to ensure they are operational and capable of handling requests. Servers that are unhealthy or underperforming can be removed from the load balancing pool.

Conclusion

Load balancing is a critical component of scaling web applications and ensuring high availability. By implementing the right load balancing strategy, you can distribute incoming traffic efficiently, prevent server overload, and optimize resource utilization. Whether you choose a static, dynamic, or client-affinity strategy, understanding the pros and cons of each will help you make an informed decision based on your application's requirements.