Maintaining Pod Availability with ReplicaSets

Everything we have run so far was a bare pod. That works until the pod disappears. Delete it by mistake, let its node crash, or have it evicted, and the application is simply gone until a human notices and recreates it. For a real application that is not acceptable.

Kubernetes has a controller whose whole job is to stop that from happening: the ReplicaSet.

The problem, demonstrated

Apply the pod and LoadBalancer Service from Creating Pod and Service in One Manifest File:

kubectl apply -f k8s-webapp.yml
kubectl get pods
kubectl get svc

Open the load balancer URL and the app loads. Now simulate an accident:

kubectl delete pod web-app-pod
kubectl get pods

No resources found in default namespace.

Refresh the browser: the page no longer loads. Nothing brings the pod back, because a bare pod has no controller watching it. No self-healing.

Clean up before moving on (the Service created an AWS load balancer):

kubectl delete -f k8s-webapp.yml --ignore-not-found

What a ReplicaSet does

A ReplicaSet makes sure that a specified number of identical pods (replicas) are running at all times.

• You declare the desired state: "3 pods with this template"
• The ReplicaSet controller (part of the controller manager) continuously compares it with the current state: how many matching pods actually exist
• Too few? It creates pods from the template. Too many? It deletes the extras

It does not matter why a pod disappeared: deleted by a person, crashed node, eviction. The ReplicaSet notices and replaces it. This is Kubernetes self-healing in practice.

Step 1: Write the ReplicaSet

vi k8s-rs.yml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: telusko-rs
  labels:
    app: telusko-rs-webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: telusko-rs-webapp
  template:
    metadata:
      labels:
        app: telusko-rs-webapp
    spec:
      containers:
        - name: webapp-container
          image: <your-dockerhub-username>/springboot-app:latest
          ports:
            - containerPort: 8080

This is the first manifest with a pod inside another object, so read it in two halves.

The ReplicaSet itself

The pod template (spec.template)

The template is a pod manifest without apiVersion and kind. Every time the ReplicaSet needs a new pod, it stamps one out from this template.

Step 2: Apply it, and fix the indentation error

kubectl apply -f k8s-rs.yml

The first attempt in the lecture failed with a BadRequest ... cannot be handled as a ReplicaSet error. Two indentation mistakes were the cause:

1. The app: line under template.metadata was written directly under metadata, without the labels: level in between. A pod template needs metadata then labels then app, exactly like a normal pod.
2. spec: under template was not indented to the same level as template.metadata. The template's spec belongs inside template, not at the ReplicaSet level.

Compare your file carefully with the one above. Template indentation is the most common ReplicaSet and Deployment mistake.

After fixing:

replicaset.apps/telusko-rs created

Step 3: Check the pods and the ReplicaSet

kubectl get pods

NAME               READY   STATUS    RESTARTS   AGE
telusko-rs-nlwl9   1/1     Running   0          40s
telusko-rs-s7mr5   1/1     Running   0          40s
telusko-rs-z4kg4   1/1     Running   0          40s

kubectl get rs        # rs is the short name for replicaset

NAME         DESIRED   CURRENT   READY   AGE
telusko-rs   3         3         3       57s

kubectl get pods -o wide shows the scheduler has spread the three pods across both worker nodes, so losing one node does not take the whole application down.

Step 4: Watch self-healing

Open a second SSH session to the EKS home server and watch the pods live:

kubectl get pods --watch

In the first session, delete one pod:

kubectl delete pod telusko-rs-nlwl9

In the watch window you see the deleted pod go to Terminating, and almost immediately a new pod (with a new random suffix) appear as Pending, ContainerCreating, then Running.

kubectl get rs

NAME         DESIRED   CURRENT   READY   AGE
telusko-rs   3         3         3       3m

Desired and current are equal again. Nobody ran a command to create the new pod; the ReplicaSet did it.

kubectl describe rs telusko-rs shows it in the Events section: SuccessfulCreate ... Created pod: telusko-rs-xxxxx.

How the ReplicaSet "owns" pods

The ReplicaSet does not track pods by name. It counts every pod in its namespace that matches its selector. Two consequences are worth knowing:

• If you create a bare pod by hand with the label app: telusko-rs-webapp, the ReplicaSet adopts it and then deletes one pod to stay at 3.
• If you remove the label from one of its pods (kubectl label pod <name> app-), the ReplicaSet no longer counts it and creates a replacement. The relabelled pod keeps running on its own, which is a handy debugging trick.

Every pod created by a ReplicaSet also has an ownerReferences entry pointing to it. See it with kubectl get pod <name> -o yaml | grep -A5 ownerReferences.

Try It Yourself

1. Create telusko-rs with 3 replicas and confirm with kubectl get rs and kubectl get pods -o wide which node each pod runs on.
2. Delete two pods at once (kubectl delete pod <a> <b>) while watching with --watch. How quickly are they replaced?
3. Create a bare pod with the label app: telusko-rs-webapp and watch what the ReplicaSet does.
4. Change matchLabels to a different value than the template labels and apply. Read the error.

Keep the ReplicaSet running if you go straight to the next lesson. Otherwise kubectl delete -f k8s-rs.yml, and delete the cluster if you are done for the day.

Official references

• Kubernetes: ReplicaSet
• Kubernetes: Controllers
• Kubernetes: Owners and Dependents

Written By: Shramik Masti