Changing StatefulSet PVCs

Limitation of changing StatefulSet PVCs comes from K8s, therefore some manual steps are required for changing StorageClass/Size of the PVC. Also, this works without disruption and data loss only if you have 3 and more replicas.

Change StorageClass or shrink PVC

In some cases you want to migrate a MariaDB or RabbitMQ to another PVC. For example if you have to change the StorageClass. In this case, follow these steps and always think of backup:

  1. In case of a PVC shrink, ensure that the PVC will still be large enough

  2. Update StorageClass/PVC Size in Yaook OpenStack CR (for example GlanceDeployment) to the new StorageClass/PVC Size

    • You will get error from infra-operator (updates for StatefulSet PVCs are forbidden in K8s)

  3. Scale down infra-operator or pause owner of the StatefulSet (e.g. MySQLService)

  4. Delete the StatefulSet but not the pods. This can be done with this command:

    kubectl get sts -o yaml <StatefulSetName> > <StatefulSetName>-backup.yaml
kubectl delete sts --cascade=orphan <StatefulSetName>

  5. Check if the pods are still there. IF NOT, DO NOT CONTINUE (in such case undo the previous steps to re-create the pods)

  6. Mark all PVCs for deletion - NO FORCE

    • For more certainty that data will not be lost but also more manual steps, mark only PVC from n-1 replica for deletion

  7. Start infra-operator or unpause owner of the StatefulSet (e.g. MySQLService)

  8. Wait until the infra-operator recreates the StatefulSet

  9. Trigger restart of the the StatefulSet if it does not restart by itself

    • If you decided in step 6, to mark only n-1 replica, do not trigger restart. Manually delete (no force) n-1 replica and wait until it will join cluster and is ready. Then you can repeat with n-2, …, 0 replica (mark pvc for deletion and delete corresponding replica)

  10. Wait until restart has completed

  11. Success

Extend PVC size

For increasing the volume size (this works at least for StorageClasses supporting volume expansion), follow these steps:

  1. Increase size in Yaook OpenStack CR (for example GlanceDeployment)

    • You will get error from infra-operator (updates for StatefulSet PVCs are forbidden in K8s)

  2. Update the PVC size for all the replicas of the STS in the K8s manually

  3. Delete the StatefulSet but not the pods. This can be done with this command:

    kubectl get sts -o yaml <StatefulSetName> > <StatefulSetName>-backup.yaml
kubectl delete sts --cascade=orphan <StatefulSetName>

  4. Wait until the infra-operator recreates the StatefulSet

  5. Trigger restart of the the StatefulSet if needed

  6. Success