Recovering a Persistent Volume in Kubernetes

This is a quick one post that came out of the past week. I had the database for this website stored on a DO volume attached to my Kubernetes cluster. I wanted to wipe out the helm install and redo it to switch from Gitlab auto devops (more later)- but I didn't want to lose my data. Turns out Kubernetes has three types of Persistent Volumes- "Reclaim", "Delete", and "Recycle".

Delete and Recycle both wind up deleting your data when the volume is deleted. I had mine set as "Reclaim" though. So, I updated my Deployment PVC to use a label selector, and added the label to the volume. However, the PVC still couldn't pick up the volume. It turns out that "Reclaim" doesn't delete the volume- but also doesn't reuse it. The volume goes into "Released" stage, which is just a flag telling you that kubernetes is done with this volume. The idea is that it stays around so that you can manually intervene. The correct way to reuse a volume is to use `kubectl get pv/myvolume -o yaml` to get a manifest. Strip out the parts that are added by the system. Delete the volume, and then apply your saved manifest. A volume with "Reclaim" as the policy will not actually delete the underlying storage when deleted. The new volume will have the same reference, so will pick the underlying storage back up. Since this new volume hasn't been bound, your PVC can now pick it up and continue where it left off.

The final thing I had to do was deal with passwords. My Postgres container generates a password on first launch, but the saved volume already had a password. To get around this I added an init container to the postgres pod to update the password to the newly generated one (you can use psql on localhost without a password).