Extend Kubernetes Persistent Volumes

Written by Pim on Friday March 9, 2018 - Comment - Permalink
Categories: devops, technology, howto - Tags: kubernetes, openstack, persistent-storage

Persistent volumes are immutable after creation in Kubernetes. Extending existing persistent volumes isn't possible in Kubernetes itself, but we can extend the volume outside Kubernetes by mounting a volume to a Linux host. In the Linux host, we'll use the resize2fs command to extend the volume, after which we will mount the volume again to the Kubernetes cluster.

I'm using an OpenStack provider and I have configured the OpenStack Kubernetes integration. This allows me to claim persistent volumes dynamically. Each Persistent Volume Claim (PVC) gets a unique identifier (UUID) and is prefixed with pvc- in the OpenStack volume overview. So, before we can extend the volume, we have to figure out which PVC is the correct one. This can be done using kubectl :

$ kubectl get pvc

Because you can't resize mounted volumes, we have to unmount the volume first. To do this, you have to scale down the pod in Kubernetes, which will lead to unmounting the volume from the host. Extend the volume size using the OpenStack CLI or OpenStack Web UI. Then mount the volume to a Linux host (can be a default CentOS machine) and login to this machine.

Check if the disk is mounted to the host and has the new size.

$ fdisk -l
$ lsblk

In my case, the disk is available as /dev/sdb

Mount the disk to a volume:

$ mount /dev/sdb /mnt/disk

Check the size, which will be the "old" size:

$ df -h

Extend the filesystem using resize2fs:

$ resize2fs /dev/sdb

And re-check the size. If everything went OK the filesystem has the new size:

$ df -h

Unmount the disk in the Linux host and in OpenStack. Then scale up the Kubernetes pod to re-attach the extended PVC to the pod.

$ unmount /mnt/disk

Note: The PVC capacity won't update! So when you had 20 GB configured when creating the PVC, it will stay 20 GB even if the filesystem is 50 GB!