The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › Deployment and operations

Kubernetes self-healing

When a container dies, what starts it again?

In Kubernetes, you write down the state you want, such as "run 3 copies of this application". Kubernetes keeps watching the actual state, and when it differs from the desired state, it corrects it by itself. Replacing a dead Pod with a new one without anyone stepping in, as happens here, is called self-healing.

Checked against documentation and standards · The official Kubernetes documentation

Concepts in this lab

Kubernetes

A platform that automates placing and managing containers across several servers. You declare a desired state (for example, run 3 copies of this application), and it keeps adjusting so that the actual state matches it. It restarts containers that die, adjusts their number according to load, and replaces them with a new version step by step.

Pod, Deployment, and Service

The basic building blocks of Kubernetes. A Pod is a group of one or more containers that run together and is the smallest unit of deployment. A Deployment manages how many Pods to keep and how to replace them with a new version. A Service provides an unchanging access address for Pods that change often, and it distributes requests among them.

Health check (liveness, readiness)

A periodic check of whether a server is healthy. Liveness checks whether the process is alive, so that it is restarted if there is a problem. Readiness checks whether the server is ready to take requests, so that traffic is not sent to a server that is not ready. This keeps requests from reaching a server while it is starting up or shutting down.

Autoscaling

A feature that automatically increases or decreases the number of servers according to metrics such as CPU utilization or request count. It adds servers when load is high and removes them when load is low to save cost. It takes time for an added server to become able to take requests, so for a surge that is known in advance it is safer to add servers ahead of time.

Containers and images

A container is a technology that packages an application with the things it needs to run and runs it in an environment isolated from other processes, while sharing the kernel of the host operating system. An image is a read-only template used to create a container, and it holds the application, libraries, and configuration. Running the same image greatly reduces problems caused by differences between environments.

Try these

• Default settings — One Pod dies. A new Pod is created automatically.
• When a node dies — The Pods that were on that node are created again on other nodes.
• When you increase the replicas — You change only the number, and Kubernetes creates as many Pods as are missing.
• The process is alive but hung — The liveness probe fails, so the container is restarted.
• It cannot take requests for a while — The readiness probe fails, so only the requests are cut off. The container is not restarted.

Running with the default settings

1. This is the normal state. The Deployment says "3 replicas", and 3 Pods are actually running, spread across two of the three nodes (Node 1 and Node 2). The Service divides requests among the three Pods.
2. The shop-b Pod is gone (for example, it was evicted because the node ran short of resources). The Service stops sending requests to this Pod. The actual number of Pods is now 2.
3. The controller compares the states. Desired: 3, actual: 2. One Pod is missing. No one has to tell it, because this comparison keeps repeating.
4. A new Pod, shop-d, is created. The scheduler picks a node with room and places the Pod there, and the kubelet on that node starts the container. The Pod that disappeared is not revived. It is replaced with a new one.
5. shop-d passed its readiness probe. The Service starts sending requests to it. There are 3 Pods again.

Result of this run

• The number of Pods is back to 3 although nobody stepped in. This is self-healing. Treat a Pod as disposable: it can die and be replaced at any time. So keep state in external storage (a volume, a DB, Redis), because files and sessions inside a container are lost when a Pod is replaced, and reach Pods through the fixed name of a Service, not through a Pod's IP.
• The controller keeps comparing the desired state with the actual state and closes the gap. This loop is the basic principle of how Kubernetes works.

Code

apiVersion: apps/v1
kind: Deployment
metadata: { name: shop }
spec:
  replicas: 3                 # desired state: always 3
  template:
    spec:
      containers:
        - name: app
          image: shop:1.0
          livenessProbe:        # if it fails, restart the container
            httpGet: { path: /actuator/health/liveness, port: 8080 }
          readinessProbe:       # if it fails, only exclude it from request targets
            httpGet: { path: /actuator/health/readiness, port: 8080 }

Assumptions of this model

This is a simple setup with 3 replicas and three nodes. The time it takes to detect a failure and recover is not shown (after a node failure is detected, it takes several minutes with the default settings before the Pods are moved). The behavior of the Deployment, the controller, and the two kinds of probes follows the explanations in the official Kubernetes documentation. It is not the result of running on a real cluster.

Previous lab: Containers and virtual machines · Next lab: Averages and percentiles