Categories
DevOps

Kubernetes StatefulSets Demo

Today, I’ll demo Kubernetes StatefulSets. We’ll see what StatefulSets are, why to use them and how to create them.

Table Of Contents
  1. Introduction
  2. Motivation
  3. Kubernetes StatefulSets Demo
  4. Summary

Introduction

What StatefulSets are? First of all, StatefulSets are Kubernetes workloads similar to Kubernetes Deployments. Hence, there could be multiple replicas in a StatefulSet. In addition, as a word stateful suggests, these replicas are of some stateful app. Let’s see why we need StatefulSet.

Motivation

Simply put, Kubernetes StatefulSets provide means for scaling stateful applications while associating each replica to the specific storage. StatefulSets also provide ordered scaling, scheduling and rolling update of stateful apps. I’ll show these in the demo below. Examples of stateful applications include databases, message buses or any app which writes to some storage.

Kubernetes StatefulSets Demo

We have already seen an example of a stateful app when we explored RabbitMQ cluster as a single Docker Swarm service. Let’s now see a demo of RabbitMQ StatefulSet on a cloud Kubernetes cluster.

Demo Prerequisites

I’ll use Linode’s managed Kubernetes cluster for the demo. Check out how easy it is to create Kubernetes Cluster on Linode. Linode is a cloud service provider recently purchased by Akamai. With this purchase, Akamai became a competitor in the cloud providers market. You can repeat this demo on your own Linode account. Create one and get 100$ credit using this link.

In addition to Kubernetes cluster, you’ll need kubectl and helm installed on your local machine.

Install RabbitMQ

Let’s now install RabbitMQ using helm. 

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install rabbitmq -n rabbitmq --create-namespace bitnami/rabbitmq --set service.type=LoadBalancer

Wait till all resources are running, healthy and ready. Next, access RabbitMQ management UI in order to make sure it is available. To login use credentials: 

    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace rabbitmq rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)"

RabbitMQ StatefulSet

As you can see, RabbitMQ was installed as Kubernetes StatefulSet set:

kubectl get sts  -n rabbitmq
NAME       READY   AGE
rabbitmq   1/1     6m10s
 kubectl get pod -n rabbitmq
NAME         READY   STATUS    RESTARTS   AGE
rabbitmq-0   1/1     Running   0          6m54s

StatefulSet’s pod is created with a specific name rabbitmq-0 from the pod template. The pod is associated with a specific CSI persistent volume bound to pvc data-rabbitmq-0. 

kubectl get pvc -n rabbitmq
NAME              STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
data-rabbitmq-0   Bound    pvc-175704a8e9da45ae   10Gi       RWO            linode-block-storage-retain   15m

The volume was dynamically provisioned thanks to Linode’s default storage class – linode-block-storage-retain.

Scaling RabbitMQ StatefulSet

If we scale RabbitMQ:

kubectl scale sts rabbitmq -n rabbitmq --replicas=3

And watch its pods:

watch kubectl get pod -n rabbitmq
Every 2.0s: kubectl get pod -n rabbitmq                                                                  rok: Tue Dec  6 23:28:48 2022

NAME         READY   STATUS              RESTARTS   AGE
rabbitmq-0   1/1     Running             0          19m
rabbitmq-1   1/1     Running             0          60s
rabbitmq-2   0/1     ContainerCreating   0          14s

We’ll notice that pods are scaled in a specific order and new replicas get the names rabbitmq-1, rabbitmq-2 accordingly.

If we inspect PVCs, we’ll see that they were dynamically provisioned as well:

kubectl get pvc -n rabbitmq
NAME              STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
data-rabbitmq-0   Bound    pvc-175704a8e9da45ae   10Gi       RWO            linode-block-storage-retain   22m
data-rabbitmq-1   Bound    pvc-6625fc49f66a47b7   10Gi       RWO            linode-block-storage-retain   2m8s
data-rabbitmq-2   Bound    pvc-8b1b8cf478574bb1   10Gi       RWO            linode-block-storage-retain   82s

As you might guess, each pod is using the corresponding PVC, e.g. rabbitmq-2 uses data-rabbitmq-2.

Not surprisingly, if you scale StatefulSet down, replicas go down in the opposite order. Moreover, if any replica goes down, Kubernetes will schedule a new one. The new replica will use the same storage the old replica used. The same is true for rolling upgrades.

These consistent replicas naming convention and association with the storage are very important. Especially, if you want to configure RabbitMQ cluster as a single service.

RabbitMQ Headless Service

You may rightfully ask, what is this rabbitmq-headless service and why was it created for RabbitMQ StatefulSet.

kubectl get svc -n rabbitmq | grep headless
rabbitmq-headless   ClusterIP      None           <none>            4369/TCP,5672/TCP,25672/TCP,15672/TCP                           37
m

Headless service serves clients of a StatefulSet (stateful app). Such clients usually don’t want dumb load-balancing as clusterIP Kubernetes service does thanks to kube-proxy. They might want to connect or be aware of all stateful app replicas at once, do some smart routing e.g. write to db controller, read from db secondaries, etc… That’s where Kubernetes headless service fits in. All its pods register to Kubernetes dns. Service clients either use dns names of the pods (by dns lookup of the headless service) or put headless service name to the connection string of a db/message bus api client which will perform dns lookup.

Let’s resolve dns name of the headless service in an ephemeral container and convince ourselves that it resolves to ip addresses of individual pods:

$ kubectl debug rabbitmq -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 -n rabbitmq

$ root@rabbitmq:/# nslookup rabbitmq-headless
Server:         10.128.0.10
Address:        10.128.0.10#53

Name:   rabbitmq-headless.rabbitmq.svc.cluster.local
Address: 10.2.0.8
Name:   rabbitmq-headless.rabbitmq.svc.cluster.local
Address: 10.2.0.9
Name:   rabbitmq-headless.rabbitmq.svc.cluster.local
Address: 10.2.0.7

Kubernetes StatefulSet Important Facts

While Kubernetes StatefulSet does a great job for stateful apps developers and users there are still some caveats you need to be aware about:

  • StatefulSet’s persistent volumes have to be deleted when they are no longer needed, e.g. after scale down or stateful app’s deletion.
  • As no one does that for us, the cluster of a StatefulSet app (e.g. db, message bus) has to be configured e.g. either in init containers, operator, sidecar containers, etc…

Summary

That’s it about Kubernetes StatefulSets. As always, feel free to share.

Recommended Kubernetes courses on Pluralsight:

Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription)

Recommended Kubernetes books on Amazon.