The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › Observability

Averages and percentiles

The average response time looks fine, so why do users complain that it is slow?

The response times of 1,000 requests were collected. Most take about 0.1 seconds, but a few are very slow. The average is the sum of all the values divided by their number, and a percentile is the value at a given percentage point when the values are lined up in order (p99 is the value at the 99% point). This lab shows the same data as an average and as percentiles, and which of them reveals the slow requests.

Checked against real execution · Checked against a separate calculation of the same data in Java

Concepts in this lab

TPS and response time percentiles

TPS is the number of requests (transactions) a system handles in 1 second. For response time, the average hides the few slow requests, so you look at percentiles. A p95 of 1 second means that 95% of requests are handled within 1 second, and p99 is the same measure taken at 99%. The few slow requests shape the user experience, so manage p95 and p99 together.

Load testing

Testing that artificially sends a volume of requests similar to real traffic to find out how much the system can withstand and which part becomes the problem first when it reaches its limit (the bottleneck). It is used before going live to learn the capacity the system can handle and to measure the effect of improvements. Tools such as JMeter, nGrinder, and k6 are used.

APM

A tool that monitors the performance of an application. It collects and shows the response time of each request, which methods or queries took the time, the errors that occurred, and more. In Java, it is usually applied as an agent, with no code changes. Examples are Pinpoint, Scouter, and Datadog.

Observability (logs, metrics, traces)

The ability to understand the internal state of a system from its external outputs alone. Logs are records of individual events, metrics are values aggregated as numbers, such as request counts and response times, and traces are records of the path one request takes through several services and the time spent in each segment. You need all three together to detect problems and find their causes.

Try these

• Default settings — 2% take 3s. The average looks fine at 0.16s, but p99 is 3s.
• When there are no slow requests — The average, the median, and p99 are almost the same.
• When only 0.5% are slow — Not even p99 catches them. You have to look at the maximum or p99.9 to see them.
• When 6% are slow — p95 is the first to jump.
• 20% take 10s — The average goes above 2s, but not a single user received a response like that.

Running with the default settings

1. The response times of 1,000 requests were collected. Of these, 980 took 0.08s to 0.12s, and 20 took 3s.
2. Average: adding up all the response times and dividing by 1,000 gives 0.16s. There are far more fast requests, so the slow ones are buried in the average. And not a single request actually got its response in 0.16s.
3. p50 (median): when the response times are lined up from fastest to slowest, the 500th value is 0.10s. It means "half of the requests finished within this time". This is the speed a typical user experiences.
4. p95: the 950th value is 0.12s. It means "95% of the requests finished within this time, and the other 5% were slower".
5. p99: the 990th value is 3.00s. It means "99% of the requests finished within this time, and only the slowest 1% were slower". The slow requests that the average could not show appear here.
6. The maximum is 3.00s, and p99.9 (the 999th value) is 3.00s. If there are 1,000,000 requests a day, about 20,000 of them would take 3s each at this ratio.

Result of this run

• The average hides a small number of slow requests. Response times follow a skewed distribution, where most are fast and only a few are very slow, so the average does not represent the "typical experience". That is why performance goals are set with percentiles, such as "p99 is 1s or less", instead of "an average of 0.2s".
• A user looking at one screen triggers several API calls, so the chance of hitting a slow one among them is higher than the share of slow individual requests. That is why the delay in the tail decides how fast the service feels.

Code

// Micrometer: also record percentiles of the response time
Timer.builder("http.server.requests")
    .publishPercentiles(0.5, 0.95, 0.99)
    .register(registry);

# application.yml
management.metrics.distribution.percentiles:
  http.server.requests: 0.5, 0.95, 0.99

Assumptions of this model

This is fixed, made-up data of 1,000 requests. Percentiles were calculated by lining the values up from smallest to largest and taking the value at that rank (the nearest-rank method). Real monitoring tools do not store every value but approximate with counts per range, so there is a small error, and averaging the percentiles of several servers is not a correct calculation. The average and the percentiles were calculated separately in Java from the same data, and they were confirmed to match the values on the screen.

Previous lab: Kubernetes self-healing · Next lab: Distributed tracing