Over the last couple weeks, I was experimenting with some cloud tools. The end result is that I moved this site from being hosted on DOKS using terraform to apply helm charts, to being hosted on a Rancher owned K8s cluster, and apps being deployed using Argocd + Kustomize. I ... would NOT recommend this for someone looking to host their blog lol (static site generators for the win!), but I feel good about the end result as a Devops professional. Just wanted to share some observations.

Rancher

I started this mainly to play with Rancher. I actually used it once before, in 2015. I saw Darren, their chief architect, give a talk presentation about what they were working on, and was blown away by the overlay networking they had built out. This was before anything like that had landed in the actual Docker distribution. It didn't feel great at the time and ECS was so easy that I laid off it, but had positive thoughts. Last month I got to see a demo of their most recent product though, and it seemed perfect for someone looking to run many templated K8s clusters. So I decided to make it happen.

I used Terraform to bring up a rancher server. Honestly, I am not sure how I would approach this in production, but for this purpose its a single node with the server started from a user-data script (you can just use a single `docker run` command). I especially appreciated that it has ACME handling built in, so long as I assigned the DNS name to the server, it just got a cert. Once the server is up, there is a rancher2 terraform provider. This allowed me to configure the things I needed. In this case, it was adding a couple helm repos to the Rancher system, bringing up a K8s cluster, and installing some "apps" into it. The apps are actually just helm charts that Rancher handles installing into the target cluster. For this I installed cert-manager, external-dns, and argocd (Rancher comes with nginx-ingress built in).

I did have some pain. This was using RKE, which is Rancher's custom Kubernetes management tool. It brings servers up, installs components on them, and networks everything together. I had some issues with Digital Ocean though. It turns out that DO nodes have names that are NOT DNS routable. This is sorta fine most of the time, but any system that depends on being able to contact the host has problems. In my case I wanted to install elastic stack monitoring, and metricbeat not being able to hit the hostname was a blocker. Eventually though I found the combination of settings that got everything working correctly. There is a lot to desire in the Kubernetes documentation around what the various components do (such as kubelet), what their APIs are (I still don't know), and how the certificate/networking stuff works. I may have to go read Kelsey Hightower's book at this point.

Argo-CD

Once Argo is installed, it can take over managing my apps. In this case, I used terraform to put a couple argocd "application" manifests on the server. The first installed the Elastic stack. The easiest solution for me was a single helm chart with the whole stack listed as dependent charts. The hard part was getting metricbeat working with the kubelet, but I figured out how to point it at the host IP address instead of the hostname. Finally, I installed a manifest for each of my target apps. I just pointed them at master in the respective repositories. Now, whenever I merge to master, argo notices and updates my app manifests. The reason I like it so much is that it keeps a view of what it installed, and keeps them up to date. It provides a handy way to view the containers etc that are running, and it feels more reliable then `helm update` as the last step of my CI process.

This isn't a GREAT solution. The manifests wind up getting updated before the new image is pushed, which can lead to some weird timing things. This isn't really an issue in this case, but to make it work I should probably be updating the infrastructure repo with the updated git hash at the end of my CI pipelines (as mentioned in my article on #gitops). That way nothing changes until tests pass- but for a personal website, I still feel good.

Transition

Finally, I shut off my old app. I was kind of surprised how easy it was to migrate. I brought up the new app with a new dns name, applied a DB snapshot to it, made sure it looked good, shut off the old one, and updated the DNS on the new one. Was up and running in basically no time. In production, would have been a bit different of course- I could rely on nobody changing the DB while I was messing with it, but that's not usually the case. In a real world case, I would have brought up the new postgres as a replica and promoted it to master when I was ready to kill the old app. I also would have transferred my letsencrypt cert instead of waiting for a new one while the site was down- there was a decent lag from when I switched DNS until the certs worked. For this experiment though, I got everything transferred and my old infra shut down. I was left with a cluster that I have even better control and monitoring over.