Kubernetes Probes in Combination¶
Now that we have learned how the liveness, readiness, and startup probes work independently, let's explore how they function when used together.
Probe Evaluation Order¶
- If a startup probe is provided, all other probes are disabled until the startup probe succeeds.
- There is no specific order for readiness and liveness probes in Kubernetes.
Docker Image¶
Here is the Docker Image used in this tutorial: reyanshkharga/nodeapp
We'll be using the probes
tag of the image. reyanshkharga/nodeapp:probes is a node.js application with the following features:
- The app has a 60 seconds startup delay.
- The app has a 10% chance of failure.
Note
reyanshkharga/nodeapp:probes is a node application with following endpoints:
GET /
Returns a JSON object containingHost
andVersion
GET /health
Returns the health status of the application
Here's the flowchart to illustrate the working of reyanshkharga/nodeapp:probes
app:
graph LR
A(Start) --> B{Healthy?};
B -->|True| C{{"Generate random
number in [1,10]"}};
B -->|False| D(Error Response);
C -->|Number is 9| E("Error response and
App becomes
unhealthy");
C -->|Number is not 9| F(Successful Response);
Step 1: Expose Application Using a Service¶
Let's create a LoadBalancer service to expose our application we'll create in the next step:
-
Create service:
-
Verify service:
Step 2: Create Deployment Without Any Probe¶
First, let's create a deployment without any probe and observe the behaviour of the app:
-
Create deployment:
-
Verify deployment and pods:
Once the pod starts running, you will observe that the container is marked as READY (1/1)
. At this point, the service my-service
will start sending traffic to the pod as all the containers in the pod are ready.
This situation is undesirable since the application experiences a delay of 60 seconds and cannot provide a prompt response. Also, the application might be unhealthy since it has a 10% chance of failure.
Step 3: Access Application¶
Open three seperate terminals to monitor the following:
-
Watch pods:
-
Stream logs:
-
Access application:
Hit the root endpoint again and again until you get an error. Now, when you try to access the
health
endpoint you'll see that the app is unhealthy.
Observation
- You'll see
curl: (52) Empty reply from server
error in the beginning - After 60 seconds you'll start getting response from the app
- The service keeps sending traffic to the pod even when it is unhealthy
This behaviour is undesirable since the traffic is being served from an unhealthy pod.
Step 4: Update the Deployment By Adding a Liveness Probe¶
Let's update the deployment by adding startup, readiness, and liveness probes to the container.
The updated deployment should look like the following:
The deployment will be rolled out.
Tip
For each type of probe you can choose any health check mechanism. It really depends on the application you are running and your use case.
For example let's say you want to use readiness probe to check if deployment.jar
file is present in the container at a specified location or not.
In that case your readiness probe may look something like this:
If /usr/app/deployment.jar
doesn't exist, the readiness probe would retry the health check after a specified periodSeconds
.
Step 5: Access Application Again¶
Try to access the application again:
Hit the root endpoint again and again.
You'll observe the following:
- The service doesn't send traffic to pods until the startup probe succeeds.
- The service doesn't send traffic to pods until readiness probe succeeds.
- The container is restarted when liveness probe fails.
- Livenss probe keeps checking app health and restarts the container when it becomes unhealthy.
- When a container is restarted, it undergoes startup, readiness, and liveness probes again.
Note
We currently have just a single replica in this deployment because my intention was to illustrate how a service behaves when it encounters an unhealthy pod. In a production environment, multiple pods will be available. The healthy pods will continue to handle traffic while any pods that do not pass the health probes will be taken out of the serving pool.
Clean Up¶
Assuming your folder structure looks like the one below:
Let's delete all the resources we created: