Categories
Best Practices DevOps Orchestration

Dynamic Provisioning of Kubernetes Storage

If you are a professional Kubernetes storage administrator you probably performed dynamic provisioning of Kubernetes storage and avoided creating the volumes manually. We’ll see the motivation for dynamic storage provisioning and how using storage classes serves this purpose.

If you later find this article useful take a look at the disclaimer for information on how to thank me.

Table Of Contents
  1. Introduction
  2. Kubernetes storage portability
  3. Kubernetes Storage Dynamic Provisioning Demo
  4. Summary

Introduction

We have seen how to create Kubernetes volumes in Kubernetes Volumes Introduction. These were examples of static storage provisioning. There are several problems with it. The most important one is that the storage is not portable. Portability will be a critical feature in case you want to migrate data from one cloud provider to another. So what can we do?

Kubernetes storage portability

Thanks to Kubernetes abstraction of workloads’ CPU and memory, stateless apps are deployed in the same way to all Kubernetes clusters.

What about stateful apps? App state may be lost after migrating to other Kubernetes clusters. To avoid that, we want app data portability features in place. I mean that if you define volumes inside pods or deployments and move to another storage provider you might (will) change all the volume definitions afterwards. That’s not portable. What is?

Using persistent volume claims

If apps (e.g. pods) need storage they should ask for storage using persistent volume claims. So, the pod definition will contain just a reference to PVC. Afterwards, storage team should provision persistent volumes compatible with PVCs. This way, there’s a clear abstraction of storage and responsibilities. Storage team provisions storage, app developers use it. If storage changes, only PVs and PVCs need to be reapplied. Not apps (pods).

Kubernetes Storage Static Provisioning

Storage team can create the persistent volume manually. Then if a pod claims storage using PVC, Kubernetes controller will try to bind the PVC to existing PVs. The controller tries to match PVs to PVC by comparing access mode, storage, volume mode, etc… If the match is found, claimref is stored in PV.

Kubernetes Storage Dynamic Provisioning

Kubernetes storage dynamic provisioning solves several issues which exist in static provisioning:

  • storage provisioning and volume creation are automatic. Both are performed manually by storage and Kubernetes admin respectively during static provisioning.
  • storage space may be wasted when PVC requests less storage than PV size.
  • PV is not portable. It has to be recreated manually after each migration to a different storage provider.

How does dynamic provisioning work?

We’ll give a short, very high-level overview about how Kubernetes storage dynamic provisioning works:

  • pod needs to specify PVC in its manifest which is basically a request for storage.
  • PVC deployment manifest needs to specify Kubernetes storage class it uses in order to provision the storage.
  • storage class manifests need to specify a storage provisioner with its parameters. The provisioner is responsible for creating the storage (e.g. folder on NFS mount) and PV of size corresponding to requested size in PVC. That’s dynamic provisioning of storage. In case the storage needs to be migrated to another provider, the only thing that changes is provisioner and its params. That’s storage portability.
  • if a default storage class exists, PVCs may use “” value for storage class name in order to use dynamic provisioning. Using default storage class achieves full storage portability. Nothing needs to change in the manifests when migrating to different storage (cloud) providers.

Important to know about PV/PVC

You can’t just delete PV and PVC and if they are in use. So, one can’t delete PVC if there are pods using it. Similarly, one can’t delete PV until bound PVC is removed. You wonder what happens to PV and the data if its bound PVC is deleted. It depends on the reclaim policy:

  • retain – PV will become released. Another matching PVC can bind to it.
  • delete – PV is deleted along with the storage. It’s the default policy for dynamic provisioning.
  • recycle – PV data is deleted rm -rf and the volume becomes available for reuse by matching PVCs. Note, this policy is deprecated.

Let’s see a demo which will illustrate dynamic storage provisioning.

Kubernetes Storage Dynamic Provisioning Demo

Demo Prerequisites

I assume you have Kubernetes cluster. If you don’t, install on your machine minikube, helm, kubectl.

Start Kubernetes cluster using minikube start --profile custom.

Note the important log messages after starting minikube:

    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

If you prefer, you can also repeat this demo on a managed Kubernetes cluster. Check out how easy it is to create Kubernetes Cluster on Linode. Get 100$ credit on Linode using this link. Linode is a cloud service provider recently purchased by Akamai. With this purchase, Akamai became a competitor in the cloud providers market. 

Demo

$ kubectl get pvc -n jenkins
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins   Bound    pvc-979a6522-616c-4997-8622-bbc66f649e86   8Gi        RWO            standard       50s
# let's show only important data about related to dynamic provisioning
$ kubectl describe pvc jenkins -n jenkins
Annotations:  
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
               volume.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Events:
  Type    Reason                 Age   From                                                                  Message
  ----    ------                 ----  ----                                                                  -------
  Normal  Provisioning           2m7s  k8s.io/minikube-hostpath_custom_009a768b-dbf4-4ee8-b019-4211e8cc6d3f  External provisioner is provisioning volume for claim "jenkins/jenkins"
  Normal  ExternalProvisioning   2m7s  persistentvolume-controller                                           waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
  Normal  ProvisioningSucceeded  2m7s  k8s.io/minikube-hostpath_custom_009a768b-dbf4-4ee8-b019-4211e8cc6d3f  Successfully provisioned volume pvc-979a6522-616c-4997-8622-bbc66f649e86
  • PV volume name derives from its bound PVC.
$ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
pvc-979a6522-616c-4997-8622-bbc66f649e86   8Gi        RWO            Delete           Bound    jenkins/jenkins   standard                8m55
$ kubectl describe pv pvc-979a6522-616c-4997-8622-bbc66f649e86
Name:            pvc-979a6522-616c-4997-8622-bbc66f649e86
Labels:          <none>
Annotations:     hostPathProvisionerIdentity: 5b0ded46-e8c3-455b-ba6a-07a79b93efb4
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           jenkins/jenkins
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        8Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/jenkins/jenkins
    HostPathType:  
Events:            <none>
  • Where is Jenkins data? As you can see in PV description above the data is at /tmp/hostpath-provisioner/jenkins/jenkins. Let’s verify that:
$ minikube ssh -p custom
docker@custom:~$ ls  /tmp/hostpath-provisioner/jenkins/jenkins
casc_configs                                 jenkins.model.JenkinsLocationConfiguration.xml               plugins.txt
config.xml                                   jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml  secret.key
copy_reference_file.log                      jenkins.telemetry.Correlator.xml                             secret.key.not-so-secret
hudson.model.UpdateCenter.xml                jobs                                                         secrets
hudson.plugins.git.GitTool.xml               logs                                                         updates
identity.key.enc                             nodeMonitors.xml                                             userContent
jenkins.install.InstallUtil.lastExecVersion  nodes                                                        users
jenkins.install.UpgradeWizard.state          plugins
  • Default storage class handled dynamic provisioning. The code of the provisioner lists basic actions which it performs automatically:
    • create storage folder.
    • change its permissions.
    • create Kubernetes volume representing the storage
    • delete storage of the volume (in order to support default dynamic provisioning volume ReclaimPolicy which is Delete).
  • Above actions are the meat of dynamic provisioning. Storage admins will have to perform these actions manually in case of static provisioning. 
$ kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  16m
  • Now, let’s try to delete pvc. We see that the action got stuck. 
$ kubectl delete pvc jenkins -n jenkins
persistentvolumeclaim "jenkins" deleted

Why? Because there’s a pod using the pvc. If you describe jenkins-0 pod and inspect its volumes, you’ll see:

  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins
    ReadOnly:   false

Let’s open another terminal and try to delete the pod while the deletion of jenkins PVC is running in the first terminal. 

kubectl delete pod jenkins-0 -n jenkins

We see that the deletion of pvc completed as well. Moreover, thanks to Delete ReclaimPolicy, the provisioner deleted PV and the data:

$ kubectl get pv
No resources found
$ minikube ssh -p custom
docker@custom:~$ ls  /tmp/hostpath-provisioner/jenkins/jenkins
ls: cannot access '/tmp/hostpath-provisioner/jenkins/jenkins': No such file or directory

If we describe jenkins pod, we’ll also see that Kubernetes default-scheduler failed to restart it, because persistentvolumeclaim "jenkins" not found. That’s basically data loss and downtime.

Now you’ll need to recreate PVC, for example by reinstalling the chart.

If you followed this demo on Linode, Linode’s default storage class dynamically provisioned CSI Volumes.

Summary

That’s it about Kubernetes storage dynamic provisioning. As always, feel free to share and comment. If you found this article useful, take a look at the disclaimer for information on how to thank me.

Recommended Kubernetes courses on Pluralsight:

Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription)

Recommended Kubernetes books on Amazon: