The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › High traffic

Caching and TTL

How much does a cache reduce database queries, and when do requests pile up again?

One popular product is requested continuously for 30 seconds. If the cache has the value, the request does not go to the database. If it does not, the request reads the value from the database and stores it in the cache (cache-aside). A value disappears from the cache when its TTL passes.

Conceptual model · A model that shows how the behavior works

Concepts in this lab

Cache: Storing frequently used data or computed results ahead of time in a place that can be accessed quickly, and reusing them. Responses get faster and the load on the source drops, because the source (usually a database) is not queried every time. In return, you must manage the problem of stale values that remain when the source changes.
Cache-aside pattern: The most common approach, in which the application manages the cache directly. To read data, the application checks the cache first. If the data is not there, it reads from the database, stores the value in the cache, and returns it. To change data, it updates the database and deletes the matching cache entry.
TTL (time to live): The length of time for which data is valid. If you set it when you put a value in a cache, the value is deleted automatically after that time passes. It is a basic safeguard that stops a value that no longer matches the source from staying in the cache forever.
Cache stampede: A situation in which, at the moment a heavily read cache entry expires, many requests that arrive at the same time all fail to find the value in the cache and query the database at once. The sudden load on the database can lead to an outage. It is mitigated by locking so that only one request queries the source, or by spreading out the expiration times.
Redis: A key-value store that keeps data in memory. It provides not only strings but also data structures such as lists, hashes, sets, and sorted sets. It is very fast because it works in memory, and it is used for caches, session stores, distributed locks, leaderboards, rate limiting, and more. It runs commands one after another on a single thread.
Distributed lock: A lock that lets only one process at a time perform a specific task among processes that run on several servers. Java's synchronized is valid only inside one server, so with several servers it is implemented by placing a "lock marker" in a store that all of them access (such as Redis). It is given an expiration time so that it is released even if the side that took it runs into a problem.

Try these

Default settings — TTL 10s. Each time the value expires, several requests go to the database at the same time.
Without a cache — Every request queries the database.
Many requests and a slow database — At the moment of expiration, dozens of requests hit the database at the same time (a cache stampede).
Same situation with the lock — Only one request goes to the database, and the others wait for its result.
Shorten the TTL to 5s — The data is fresher, but there are more database queries.
A long TTL — The database is queried only when the cache is first filled. But a change in the source goes unnoticed for a long time.

Running with the default settings

0.0s: The cache has no value (the first request). The request goes to the database to query it. It takes 1s for the result to come back.
1.0s: The value read from the database was stored in the cache (TTL 10s). Before it was stored, 5 requests, starting at 0.0s, all failed to find the value in the cache and each queried the database. The other requests store the same value again as their queries finish, so the TTL is counted from the last store (1.8s).
11.8s: The TTL has passed, so the value in the cache was removed. From now on, requests that arrive will not find the value in the cache.
11.8s: The cache has no value (right after expiration). The request goes to the database to query it. It takes 1s for the result to come back.
12.8s: The value read from the database was stored in the cache (TTL 10s). Before it was stored, 5 requests, starting at 11.8s, all failed to find the value in the cache and each queried the database. The other requests store the same value again as their queries finish, so the TTL is counted from the last store (13.6s).
23.6s: The TTL has passed, so the value in the cache was removed. From now on, requests that arrive will not find the value in the cache.
23.6s: The cache has no value (right after expiration). The request goes to the database to query it. It takes 1s for the result to come back.
24.6s: The value read from the database was stored in the cache (TTL 10s). Before it was stored, 5 requests, starting at 23.6s, all failed to find the value in the cache and each queried the database. The other requests store the same value again as their queries finish, so the TTL is counted from the last store (25.4s).

Result of this run

At the moment of expiration, all the requests that arrived before the cache was filled again (during 1s) went to the database. The more requests there are and the slower the database query is, the bigger the pile-up. This is a cache stampede. You reduce it with a lock that lets only one request query the database, or by refreshing the value before it expires. When many keys expire at the same time, give each key a slightly different TTL.
The longer the TTL, the fewer the database queries, but even when the source data changes, the cache keeps returning the old value for up to the TTL. How old a value you can accept is the criterion for choosing the TTL.

Code

public Product getProduct(long id) {
    String key = "product:" + id;
    Product cached = redis.get(key);
    if (cached != null) return cached;          // cache hit

    Product p = productRepository.findById(id); // database query (1s)
    redis.set(key, p, Duration.ofSeconds(10));  // TTL 10s
    return p;
}

Assumptions of this model

Only one key (one popular product) is handled, and requests are assumed to arrive at regular intervals. The cache response time is set to 0.01s. The eviction policy, which removes old keys when the cache is full, is not covered here. With the lock, a waiting request is simplified to receive the value from the cache at the moment the query finishes. The numbers are a model that shows how the behavior works, and they are not values measured on a real Redis.

Previous lab: Load balancers and scale out · Next lab: Redis single-threaded execution

Concepts in this lab

What this lab shows

Settings

Predict the result before running

Code

Diagram

Run comparison