Dev Learning Lab

The labs on this site need JavaScript to run. The same content is written out below.

Learn › Java › Threads and concurrency

Thread pool

What changes when you add more threads

A thread pool reuses the threads it has created, up to a set number, instead of creating and discarding a thread for every request. When a request comes in, one thread handles it from start to finish. When all the core threads are busy, requests wait in a queue; when the queue is full, more threads are created up to the maximum; and requests beyond that are rejected. This lab shows how waiting and rejection change with the number of threads and the size of the queue.

Checked against real execution · Checked against the results of actually running a ThreadPoolExecutor

Concepts in this lab

Thread pool
This structure keeps a fixed number of threads and reuses them, so that threads are not created and destroyed every time they are needed. It reduces the cost of creating threads, and by limiting the number of threads that run at the same time, it prevents excessive memory use and context switching. Tasks to be processed pile up in a queue, and an idle thread takes one and runs it. Tomcat's request handling and a DB connection pool follow the same idea.
How ThreadPoolExecutor handles a new task
Java's thread pool implementation handles an incoming task in the following order. If fewer threads than corePoolSize are running, it creates a new thread. Otherwise, it puts the task in the queue. If the queue is full, it creates more threads, up to maximumPoolSize. Beyond that, it rejects the task. Contrary to intuition, with a bounded queue the number of threads grows only after the queue is full (with an unbounded queue it does not grow).
Tomcat's threading model
Tomcat manages its request processing threads in a pool, and one thread takes one request and handles it until the response is finished. The number of requests it can handle at the same time is limited by the maximum number of threads (200 by default), and when all threads are in use, new requests wait. A thread stays occupied even while the request is waiting on a database or an external call.
Blocking and non-blocking
Blocking means that the calling thread is stopped until the called task finishes and cannot do other work. Non-blocking means that the call returns right away even though the task has not finished, so the thread can do other work. A typical JDBC call is a blocking call: the thread is stopped until the DB response arrives.

Try these

Running with the default settings

  1. Request 1 arrives. The pool has fewer threads than the core pool size (2), so it creates thread 1, which handles the request right away.
  2. Request 2 arrives. The pool has fewer threads than the core pool size (2), so it creates thread 2, which handles the request right away.
  3. Request 3 arrives. Every thread is busy (2 in the pool). Instead of adding a thread, the pool puts the request in the queue (waiting: 1/2).
  4. Request 4 arrives. Every thread is busy (2 in the pool). Instead of adding a thread, the pool puts the request in the queue (waiting: 2/2).
  5. Request 5 arrives. The queue is full. Only now does the pool create an additional thread, thread 3, to handle it (up to a maximum of 4).
  6. Request 6 arrives. The queue is full. Only now does the pool create an additional thread, thread 4, to handle it (up to a maximum of 4).
  7. Request 7 arrives. Every thread is busy (the maximum, 4) and the queue is also full. The request is rejected.
  8. Request 8 arrives. Every thread is busy (the maximum, 4) and the queue is also full. The request is rejected.
  9. Request 9 arrives. Every thread is busy (the maximum, 4) and the queue is also full. The request is rejected.
  10. Request 10 arrives. Every thread is busy (the maximum, 4) and the queue is also full. The request is rejected.
  11. Thread 1 finishes request 1, takes request 3 from the head of the queue, and starts handling it. Request 3 waited 1.6s.
  12. Thread 2 finishes request 2, takes request 4 from the head of the queue, and starts handling it. Request 4 waited 1.6s.
  13. Thread 3 finishes request 5. No request is waiting, so the thread becomes idle.
  14. Thread 4 finishes request 6. No request is waiting, so the thread becomes idle.
  15. Thread 1 finishes request 3. No request is waiting, so the thread becomes idle.
  16. Thread 2 finishes request 4. No request is waiting, so the thread becomes idle.

Result of this run

Code

ExecutorService pool = new ThreadPoolExecutor(
    2,                        // core pool size
    4,                        // maximum pool size
    60, TimeUnit.SECONDS,
    new ArrayBlockingQueue<>(2));     // queue capacity 2

for (int i = 1; i <= 10; i++) {
    pool.execute(() -> handle());   // each request takes 2.0s
    Thread.sleep(200);            // a request arrives every 0.2s
}

Assumptions of this model

While a request is being handled, one thread is tied up for the whole handling time (a request that waits for a slow database or external API). The model assumes that adding threads costs nothing. In reality each thread uses memory, and when more threads want the CPU than there are cores, the cost of switching between them grows. The order in which threads are assigned (core threads → queue → threads up to the maximum → rejection) is the same as the actual rule of the Java ThreadPoolExecutor, and it was checked against real execution.

Previous lab: Deadlock · Next lab: Thread count and context switching

Concepts in this lab

What this lab shows

Settings

Code


      
    


    

      
Diagram

      

      

    


    

      

      

      

    


  


  

    

      
Run comparison

      

      

    

    

      
Try these