The labs on this site need JavaScript to run. The same content is written out below.

Learn › Java › Threads and concurrency

Thread count and context switching

Adding threads does not keep making things faster

A CPU core runs only one thread at any given moment. When more threads want to run than there are cores, the operating system runs them in turn, each for a short time (context switching). Throughput then stops growing, and only the time for one task to finish gets longer. The right number of threads is completely different depending on whether the tasks mostly compute or mostly wait.

Partly checked against real execution · The trend is the same as in measurements on JDK 21 (the numbers come from the model)

Concepts in this lab

Processes and threads

A process is a running program, and the operating system gives it its own independent memory space. A thread is a flow of execution that does the actual work inside a process. The threads of one process share memory (heap, code, and data), and each has only its own stack. This makes it easy for threads to exchange data, but it also causes problems when they touch the same data at the same time.

Context switching

This is the process in which the CPU stops the thread (or process) it is running and switches to run another one. The state of the current work must be saved and the state of the next work must be loaded, and the switch itself takes time. When there are far more threads than CPU cores, the time spent on switching grows relative to the time spent on actual work.

Thread pool

This structure keeps a fixed number of threads and reuses them, so that threads are not created and destroyed every time they are needed. It reduces the cost of creating threads, and by limiting the number of threads that run at the same time, it prevents excessive memory use and context switching. Tasks to be processed pile up in a queue, and an idle thread takes one and runs it. Tomcat's request handling and a DB connection pool follow the same idea.

Blocking and non-blocking

Blocking means that the calling thread is stopped until the called task finishes and cannot do other work. Non-blocking means that the call returns right away even though the task has not finished, so the thread can do other work. A typical JDBC call is a blocking call: the thread is stopped until the DB response arrives.

Try these

• Default settings — 4 cores and 16 threads that do computation. Throughput is the same as with 4 threads, and only the time for one task to finish becomes 4 times as long.
• Threads matched to the number of cores — Throughput is the same, and every task finishes without waiting.
• Mostly waiting, 4 threads — The threads are waiting most of the time, so the CPU sits idle.
• Mostly waiting, 40 threads — While one thread waits, other threads use the CPU, so throughput becomes 10 times as high.
• Even more threads — 80 threads. The CPU is already fully used, so throughput stays the same.

Running with the default settings

1. Cores: 4. Threads: 16. Each thread computes without a break.
2. At any given moment, the average number of threads that want the CPU is 16 (16 × 100%). The number of cores is 4, so there are not enough cores. The operating system puts the threads on the cores in turn, each for a short time.
3. Each time the operating system switches threads, it saves the state of the thread that was running and loads the state of the next one. About 4 threads share each core, so each thread is stopped whenever it is not its turn. The total amount of work done stays at what the cores (4) can do, and only the time for each task to finish gets longer.
4. Result: CPU usage 100%, tasks handled per second: 40, time for one task to finish: 0.4s (0.1s when it runs alone). With the number of threads set to 4, the values become 40 per second and 0.1s per task.

Result of this run

• For tasks that mostly compute, having more threads than cores does not increase throughput. The threads share the cores, so each task just finishes later. Each thread also uses memory of its own (its stack), so a number of threads close to the number of cores is appropriate.
• A web server has far more threads for handling requests than it has cores (200 by default in Tomcat) because most of the time spent handling a request goes to waiting for a database or an external response.

Code

int cores = Runtime.getRuntime().availableProcessors();   // 4

// mostly computation: as many threads as cores is enough
ExecutorService pool = Executors.newFixedThreadPool(16);

for (Task task : tasks) {
    pool.submit(() -> resize(image));
}

Assumptions of this model

This model assumes that one task uses the CPU for 0.1s. The trend is the same as in results actually measured on JDK 21 on this PC (12 logical cores). For tasks that mostly compute, throughput was 102 tasks per second with 1 thread, 562 with 6, 768 with 12, 820 with 24, and 826 with 48, so it stopped growing near the number of cores. For tasks that mostly wait (1ms of computation + 9ms of waiting), it was 1,104 with 12 threads, 4,417 with 48, 7,871 with 120, 8,202 with 240, and 8,144 with 480, so it grew up to about 10 times the number of cores and then stopped. Even with far more threads than cores, throughput did not drop noticeably. The cost of the switching itself was too small to show up in this measurement, which is why the model does not include a drop in throughput either. The measured throughput is not exactly proportional to the number of cores because the 12 logical cores share 6 physical cores. The formula for the suitable number of threads is a widely cited guideline in the field of Java concurrency.

Previous lab: Thread pool · Next lab: Sequential and parallel calls