The labs on this site need JavaScript to run. The same content is written out below.

Learn › Networking › In front of the web server

Load balancers and scale out

How far does adding servers get you?

Scaling out means adding more servers to share the load instead of making one server bigger. A load balancer distributes requests across several servers, and each request uses the database once. This lab shows how far the throughput grows as you add servers, and which part becomes the bottleneck first.

Conceptual model · Example numbers chosen to show the relationships

Concepts in this lab

Scale up and scale out

Scaling up means raising the specifications (CPU, memory) of a single server to increase its capacity, and scaling out means adding more servers to share the load. Scaling up is simple but has limits and cost problems. Scaling out can grow flexibly, but the servers must be designed to hold no state, and you need something that distributes the requests.

Load balancer

A device that distributes incoming requests across several servers. It checks the state of the servers periodically (health check) and does not send requests to a server that has a problem. It keeps load from piling up on one server and keeps the service running even when some servers fail.

Bottleneck

The slowest part of a system, which limits the processing performance of the whole. No matter how much you improve the other parts, overall performance does not improve while this part stays the same. Performance improvement starts with finding it by measurement.

Autoscaling

A feature that automatically increases or decreases the number of servers according to metrics such as CPU utilization or request count. It adds servers when load is high and removes them when load is low to save cost. It takes time for an added server to become able to take requests, so for a surge that is known in advance it is safer to add servers ahead of time.

TPS and response time percentiles

TPS is the number of requests (transactions) a system handles in 1 second. For response time, the average hides the few slow requests, so you look at percentiles. A p95 of 1 second means that 95% of requests are handled within 1 second, and p99 is the same measure taken at 99%. The few slow requests shape the user experience, so manage p95 and p99 together.

Try these

• Default settings — 1,000 requests per second with 2 servers. The servers cannot keep up.
• Go to 3 servers (scale out) — Add one more server of the same spec.
• Double the server spec (scale up) — The number of servers stays at 2; the performance of each one goes up.
• Keep adding servers — 3,000 requests per second with 8 servers. This time the database becomes the bottleneck.
• 1 of 3 servers fails — The remaining 2 share the requests. If they have no spare capacity, they overflow together.
• Failure with only 1 server — There is no server left to accept requests, so the whole service stops.

Running with the default settings

1. Requests arrive at the load balancer at 1000 per second. Users do not know how many servers there are; they know only the one address of the load balancer.
2. The load balancer divides the requests evenly among the 2 live servers. Each server receives 500 requests per second. One server can handle 400 requests per second, so its load is 125%. This is over its capacity. The overflowing requests wait and then time out or are rejected.
3. The 800 requests per second that the servers handled go on to the database. The database can handle 2000 requests per second, so its load is 40%.
4. Result: 800 requests per second are handled and 200 are not. Bottleneck: Application server.

Result of this run

• Of 1000 requests per second, 800 were handled (200 over the limit). The load on one server is 125%, and the load on the database is 40%.
• The application servers are the bottleneck. Adding more servers (scale out) or raising their spec (scale up) fixes it. Scale out keeps working when one server dies and can be continued as far as you like, but the servers must not keep things such as login state in their own memory. Scale up needs no change in structure, but it has a limit and the cost rises quickly.

Code

# Load balancer settings (nginx example)
upstream app {
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
}
server {
    listen 80;
    location / { proxy_pass http://app; }   # passes requests to the servers in turn
}

# One server: handles 400 requests per second
# DB: handles 2000 requests per second (1 query per request)

Assumptions of this model

This model simplifies things: every request weighs the same, and one request uses one database query. The load balancer is assumed to divide requests evenly and to have no limit of its own (in practice, load balancers are also made redundant). Requests beyond capacity are only counted as "not handled"; in practice, as the load gets closer to the limit, the waiting time grows sharply first. The numbers are examples to show the relationships, not measured values.

Previous lab: TLS handshake · Next lab: Proxies and reverse proxies