Dev Learning Lab

The labs on this site need JavaScript to run. The same content is written out below.

Fundamentals › Architecture and operations

Load balancers

If you run several servers, which server does a request go to?

When there are more requests than one server can handle, you start the same server program on several machines. But it is inconvenient if the client (the side that sends requests) has to know several server addresses. So the client connects to one address only, and the load balancer that receives requests at that address spreads them across the servers. The load balancer in this lab sends requests to server A and server B in turn, and skips a server whose connection is refused. The code box is not Java code. It is a configuration that says which program runs at which address and what it does (lines that start with # are comments). Each server counts the requests it has received in its own memory and replies like "I am A, request #1". In this lab you remove the load balancer, and you stop the server A program after 3 requests, and you see how the replies the client receives change.

Checked against real execution · Ran all four combinations with an http server of Node.js 24.18.0 as server A (and B), a load balancer built by hand, and an http client, and checked against the model

Concepts in this lab

Load balancer
A device that distributes incoming requests across several servers. It checks the state of the servers periodically (health check) and does not send requests to a server that has a problem. It keeps load from piling up on one server and keeps the service running even when some servers fail.
Round robin
Round robin is a method that hands out requests to the servers in turn, one server at a time. With two servers, A and B, requests go to A, B, A, B, in that order. The NGINX documentation says this method, which distributes requests evenly across the servers, is the default and that no directive is needed to turn it on, and that you can also give each server a weight. Which server gets which request follows a fixed order, so when the servers are always in the same state, requests are spread evenly across them.
Skipping a failed server
Skipping a failed server is what a load balancer does when an error occurs while it tries to connect to a server: it does not use that server and sends the same request to the next server. In the NGINX documentation the default of proxy_next_upstream is error and timeout, and error covers an error that occurs while establishing a connection with the server, so when a connection is refused, the request is passed to the next server. Also, after max_fails failures (default 1), the server is considered unavailable for fail_timeout (default 10 seconds). Because of this, even if one server stops, a new request is handled by a live server as long as other servers remain (a request that was already being handled may fail).
Connection refused error
A connection refused error is the error that the sender gets when it asks for a TCP connection but no program has that port open and is waiting. In Node.js, the error event comes instead of the connect event, and the error code appears as ECONNREFUSED (the Node.js errors documentation describes ECONNREFUSED as "Connection refused: No connection could be made because the target machine actively refused it", and the net documentation says that if there is a problem with the connection, error comes instead of connect). Because the connection was not opened, not a single letter of text is sent. The TCP standard says to send a reset signal (RST) to a request that comes with no connection (RFC 9293 section 3.5.2).

Try these

Running with the default settings

  1. The servers start. Two copies of the same server program start separately as server A and server B. Each one counts the requests it has received in its own memory, and when a request arrives it replies "I am A" or "I am B" together with that count. A load balancer starts in front of them. The client does not know how many servers there are. It knows only one address, localhost:8080. The program that receives requests at this address is the load balancer.
  2. The client sends request #1, GET /. What receives the request at address 8080 is not a server but the load balancer. The load balancer passes this request to server A, the first server in its list.
  3. Server A adds one to the number of received requests it keeps in its own memory and replies "I am A, request #1". The load balancer returns this reply to the client unchanged. The text the client receives says which server replied.
  4. The client sends request #2, GET /. The load balancer sends requests in turn, so this time it is the turn of server B. Server B adds one to the count it keeps in its own memory and replies "I am B, request #1", and the load balancer returns this reply. Server A has already replied once, but server B's count starts at "request #1". Each server counts the requests it has received separately, in its own memory.
  5. The client sends request #3, GET /. The load balancer sends requests in turn, so this time it is the turn of server A. Server A adds one to the count it keeps in its own memory and replies "I am A, request #2", and the load balancer returns this reply.
  6. The client sends request #4, GET /. The load balancer sends requests in turn, so this time it is the turn of server B. Server B adds one to the count it keeps in its own memory and replies "I am B, request #2", and the load balancer returns this reply.
  7. The client sends request #5, GET /. The load balancer sends requests in turn, so this time it is the turn of server A. Server A adds one to the count it keeps in its own memory and replies "I am A, request #3", and the load balancer returns this reply.
  8. The client sends request #6, GET /. The load balancer sends requests in turn, so this time it is the turn of server B. Server B adds one to the count it keeps in its own memory and replies "I am B, request #3", and the load balancer returns this reply.
  9. The 6 requests are finished. Server A replied 3 times and server B replied 3 times, and 0 requests failed. The count of received requests kept by server A is 3, and the count kept by server B is 3.

Result of this run

Code

# Client: 6 requests to http://localhost:8080/
# 8080: load balancer (sends to server A and server B in turn)
# Server A (9001), server B (9002): reply "I am A" and "I am B"
#   The reply also includes the number of requests the server has received

Assumptions of this model

Server A (and server B) is an http server of Node.js 24.18.0, and each server counts the requests it has received in its own variable and replies like "I am A, request #1". The load balancer is not a product such as NGINX but one built by hand as an http server of the same Node.js. For each request it moves the starting position in the server list by one (A, B, A, B...), forwards the request to that server on a new connection, skips a server whose connection is refused and sends the same request to the next server, and does not try a server again after it has refused once. The client sent a GET / request 6 times in a row with the http module, on a new connection for each request. Stopping server A was done by closing that server's listening socket and its open connections instead of ending the process, and after it was closed, a connection from the client or the load balancer gave ECONNREFUSED (a connection refused error). I ran all four combinations and checked against the model the number of replies from server A and server B, the number of failed requests, which server replied to each of the 6 requests and as which request number, the counts the servers kept in memory, the requests and servers for which the load balancer had its connection refused, and the request text the server received. I also checked for each combination that the code on the screen has the same content as that setup. The code on the screen is a description of that setup, not the program that was run, and 8080, 9001, and 9002 on the screen are example addresses; the real runs used free ports. These are behaviors observed in this implementation. What I checked against documentation is RFC 9110 section 3.7 (the definition that a gateway, that is a reverse proxy, is an intermediary that forwards received requests to another server or to several servers inside) and the NGINX documentation (ngx_http_upstream_module at nginx.org: requests are distributed among the servers by weighted round robin by default, and max_fails with a default of 1 and fail_timeout with a default of 10 seconds make a failed server be considered unavailable for that time; ngx_http_proxy_module: the default of proxy_next_upstream is error timeout, and error includes an error that occurred while establishing a connection with the server). NGINX is not installed on the PC used to write this lab, so I did not run NGINX. The load balancer in this lab was only built to act in the same spirit as NGINX, and it was not checked against the real behavior of NGINX. Not covered: the case where the load balancer itself stops, the case where a stopped server starts again and comes back, health checks that check the state of servers periodically, servers with different processing power or weights, sending requests from the same user to the same server, and the limit a server can bear when there are very many requests. How far throughput grows when you add servers is covered in the in-depth lab "Load balancers and scale out", how to replace servers one at a time in a deployment is covered in "Deployment strategies", and automatically starting a dead server again is covered in "Kubernetes self-healing".

Previous lab: Calling another server · Next lab: Caches

Concepts in this lab

What this lab shows

Settings

Code


      
    


    

      
Diagram

      

      

    


    

      

      

      

    


  


  

    

      
Run comparison

      

      

    

    

      
Try these