If you are a professional Kubernetes storage administrator you probably performed dynamic provisioning of
Kubernetes storage and avoided creating the volumes manually. We’ll see the motivation for dynamic storage provisioning and how using storage classes serves this purpose.
We have seen how to create Kubernetes volumes in Kubernetes Volumes Introduction. These were examples of static storage provisioning. There are several problems with it. The most important one is that the storage is not portable. Portability will be a critical feature in case you want to migrate data from one cloud provider to another. So what can we do?
Kubernetes storage portability
Thanks to Kubernetes abstraction of workloads’ CPU and memory, stateless apps are deployed in the same way to all Kubernetes clusters.
What about stateful apps? App state may be lost after migrating to other Kubernetes clusters. To avoid that, we want app data portability features in place. I mean that if you define volumes inside pods or deployments and move to another storage provider you might (will) change all the volume definitions afterwards. That’s not portable. What is?
Using persistent volume claims
If apps (e.g. pods) need storage they should ask for storage using persistent volume claims. So, the pod definition will contain just a reference to PVC. Afterwards, storage team should provision persistent volumes compatible with PVCs. This way, there’s a clear abstraction of storage and responsibilities. Storage team provisions storage, app developers use it. If storage changes, only PVs and PVCs need to be reapplied. Not apps (pods).
Kubernetes Storage Static Provisioning
Storage team can create the persistent volume manually. Then if a pod claims storage using PVC, Kubernetes controller will try to bind the PVC to existing PVs. The controller tries to match PVs to PVC by comparing access mode, storage, volume mode, etc… If the match is found,
claimref is stored in PV.
Kubernetes Storage Dynamic Provisioning
Kubernetes storage dynamic provisioning solves several issues which exist in static provisioning:
- storage provisioning and volume creation are automatic. Both are performed manually by storage and Kubernetes admin respectively during static provisioning.
- storage space may be wasted when PVC requests less storage than PV size.
- PV is not portable. It has to be recreated manually after each migration to a different storage provider.
How does dynamic provisioning work?
We’ll give a short, very high-level overview about how Kubernetes storage dynamic provisioning works:
- pod needs to specify PVC in its manifest which is basically a request for storage.
- PVC deployment manifest needs to specify Kubernetes storage class it uses in order to provision the storage.
- storage class manifests need to specify a storage provisioner with its parameters. The provisioner is responsible for creating the storage (e.g. folder on NFS mount) and PV of size corresponding to requested size in PVC. That’s dynamic provisioning of storage. In case the storage needs to be migrated to another provider, the only thing that changes is provisioner and its params. That’s storage portability.
- if a default storage class exists, PVCs may use “” value for storage class name in order to use dynamic provisioning. Using default storage class achieves full storage portability. Nothing needs to change in the manifests when migrating to different storage (cloud) providers.
Important to know about PV/PVC
You can’t just delete PV and PVC and if they are in use. So, one can’t delete PVC if there are pods using it. Similarly, one can’t delete PV until bound PVC is removed. You wonder what happens to PV and the data if its bound PVC is deleted. It depends on the reclaim policy:
- retain – PV will become released. Another matching PVC can bind to it.
- delete – PV is deleted along with the storage. It’s the default policy for dynamic provisioning.
- recycle – PV data is deleted
rm -rfand the volume becomes available for reuse by matching PVCs. Note, this policy is deprecated.
Let’s see a demo which will illustrate dynamic storage provisioning.
Kubernetes Storage Dynamic Provisioning Demo
I assume you have
Kubernetes cluster. If you don’t, install on your machine
Kubernetes cluster using
minikube start --profile custom.
Note the important log messages after starting
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass
If you prefer, you can also repeat this demo on a managed Kubernetes cluster. Check out how easy it is to create Kubernetes Cluster on Linode. Get 100$ credit on Linode using this link. Linode is a cloud service provider recently purchased by Akamai. With this purchase, Akamai became a competitor in the cloud providers market.
- Install Jenkins helm chart
- let’s explore storage related objects:
- jenkins PVC. Thanks to default storage class used by PVC,
minikubehostPath provisioner created storage for Jenkins datal. The provisioner also created hostPath volume of the requested size and bound it to PVC.
- jenkins PVC. Thanks to default storage class used by PVC,
$ kubectl get pvc -n jenkins NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE jenkins Bound pvc-979a6522-616c-4997-8622-bbc66f649e86 8Gi RWO standard 50s # let's show only important data about related to dynamic provisioning $ kubectl describe pvc jenkins -n jenkins Annotations: volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath volume.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 2m7s k8s.io/minikube-hostpath_custom_009a768b-dbf4-4ee8-b019-4211e8cc6d3f External provisioner is provisioning volume for claim "jenkins/jenkins" Normal ExternalProvisioning 2m7s persistentvolume-controller waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator Normal ProvisioningSucceeded 2m7s k8s.io/minikube-hostpath_custom_009a768b-dbf4-4ee8-b019-4211e8cc6d3f Successfully provisioned volume pvc-979a6522-616c-4997-8622-bbc66f649e86
- PV volume name derives from its bound PVC.
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-979a6522-616c-4997-8622-bbc66f649e86 8Gi RWO Delete Bound jenkins/jenkins standard 8m55 $ kubectl describe pv pvc-979a6522-616c-4997-8622-bbc66f649e86 Name: pvc-979a6522-616c-4997-8622-bbc66f649e86 Labels: <none> Annotations: hostPathProvisionerIdentity: 5b0ded46-e8c3-455b-ba6a-07a79b93efb4 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath Finalizers: [kubernetes.io/pv-protection] StorageClass: standard Status: Bound Claim: jenkins/jenkins Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 8Gi Node Affinity: <none> Message: Source: Type: HostPath (bare host directory volume) Path: /tmp/hostpath-provisioner/jenkins/jenkins HostPathType: Events: <none>
- Where is Jenkins data? As you can see in PV description above the data is at
/tmp/hostpath-provisioner/jenkins/jenkins. Let’s verify that:
$ minikube ssh -p custom [email protected]:~$ ls /tmp/hostpath-provisioner/jenkins/jenkins casc_configs jenkins.model.JenkinsLocationConfiguration.xml plugins.txt config.xml jenkins.security.apitoken.ApiTokenPropertyConfiguration.xml secret.key copy_reference_file.log jenkins.telemetry.Correlator.xml secret.key.not-so-secret hudson.model.UpdateCenter.xml jobs secrets hudson.plugins.git.GitTool.xml logs updates identity.key.enc nodeMonitors.xml userContent jenkins.install.InstallUtil.lastExecVersion nodes users jenkins.install.UpgradeWizard.state plugins
- Default storage class handled dynamic provisioning. The code of the provisioner lists basic actions which it performs automatically:
- create storage folder.
- change its permissions.
- create Kubernetes volume representing the storage
- delete storage of the volume (in order to support default dynamic provisioning volume
ReclaimPolicywhich is Delete).
- Above actions are the meat of dynamic provisioning. Storage admins will have to perform these actions manually in case of static provisioning.
$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) k8s.io/minikube-hostpath Delete Immediate false 16m
- Now, let’s try to delete pvc. We see that the action got stuck.
$ kubectl delete pvc jenkins -n jenkins persistentvolumeclaim "jenkins" deleted
Why? Because there’s a pod using the pvc. If you describe
jenkins-0 pod and inspect its volumes, you’ll see:
jenkins-home: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: jenkins ReadOnly: false
Let’s open another terminal and try to delete the pod while the deletion of
jenkins PVC is running in the first terminal.
kubectl delete pod jenkins-0 -n jenkins
We see that the deletion of pvc completed as well. Moreover, thanks to
ReclaimPolicy, the provisioner deleted PV and the data:
$ kubectl get pv No resources found $ minikube ssh -p custom [email protected]:~$ ls /tmp/hostpath-provisioner/jenkins/jenkins ls: cannot access '/tmp/hostpath-provisioner/jenkins/jenkins': No such file or directory
If we describe jenkins pod, we’ll also see that Kubernetes
default-scheduler failed to restart it, because
persistentvolumeclaim "jenkins" not found. That’s basically data loss and downtime.
Now you’ll need to recreate PVC, for example by reinstalling the chart.
If you followed this demo on Linode, Linode’s default storage class dynamically provisioned CSI Volumes.
That’s it about
Kubernetes storage dynamic provisioning. As always, feel free to share and comment.
- Become a Certified Kubernetes Administrator (CKA)!
- Become a Certified Kubernetes Application Developer (CKAD)!
- BUNDLE KUBERNETES FUNDAMENTALS & CKA CERTIFICATION (COURSE & CERTIFICATION) FOR THE BEST DEAL! $499 ONLY!
Kubernetes courses on Pluralsight:
- Deploying Stateful Applications in Kubernetes
- Configuring and Managing Kubernetes Storage and Scheduling
Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription)
Recommended Kubernetes books on Amazon: