#GitOps is an Anti-Pattern

Have been working on a project recently, helping a new team set up SDLC into a K8s cluster. Fun stuff I've done a bunch of times before. However, they didn't want to do it the way I always have- they wanted to do GitOps.

Now, I hadn't been following closely, and had actually assumed what I was doing WAS GitOps. Each commit to master results in an an artifact built and pushed into a testing environment, followed by tests, followed by promotion to the next environment... until it is running in Prod. Each commit has a pipeline so you can see how far along it is, and look at the CI system to see which commit was the last one to roll out.

Turns out, this isn't what they mean. Instead, they recommend having a separate repo for "config". You deploy to staging by updating that repo with the version of the new artifact, and deploy to production the same way. You can always look at that repo and see the current version of whatever is deployed to each environment. Okay- doesn't sound terrible, I figured I would approach it with an open mind.

I have an opinion now, and I am not a fan. Sure, it does work. But it doesn't give any benefits that I can see. For example, to promote from staging to Production, I would want to do a pull request of the last batch of changes. But how does that work? Imagine my repo has a file that says `VERSION=47`. My new deploy bumps that to `VERSION=48`. The first thing I tried was having a branch per environment, do a PR from staging to prod. Sounded great, but the mechanics of it are terrible. It was very hard to make sure not to pull staging config into the prod environment! Worse, new contributors struggled to catch up. Finally, having a developer make a git commit to deploy to staging, and then a PR to promote, was painful, and didn't allow any integration testing.

So, we decided to go to one branch, but a file per environment. This gives you that one view that you want. However, the promotion process is actually worse now- copy `VERSION=48` from `staging.yaml` into `prod.yaml` and open another PR. What happens when you mess it up? Even worse, what about SHARED config? For example, I want to experiment with changing the load balancer setup- promoting that change to prod when I'm happy is manual and error prone. Also didn't really get us integration testing.

Next I set up a Jenkins pipeline. It would check out the config repo and update `VERSION` to the new version, wait for it to deploy, and then run tests, with a final approval that would lead to it copying the changes over into prod. This is starting to get better. To improve it more, all the config was moved back to the app repo, and referred to with a git tag. so VERSION now referred both to the config files and the artifact, making it easy to ensure that a version was pointed at the config files for that specific version.

So, we had something that works- but lets look closer. We build an artifact, update deployed state, run some tests, and update deployed state again. Sounds... like every other deployment tool ever. Except- we're using github as the state store. Git is great at a lot of things, but it wouldn't be in the top 50 of my choices for a database. It feels like literally any other state store would accomplish the same thing- better. Besides, we're back to the existing toolsets already anyway.

Argocd is a good tool for APPLYING config. It feels better than helm apply in a jenkins script. Right now though, I think the right thing to do is to just point the argo application at the app repo, with the deployed environment controlled by a tag. Run "staging" on "master" and "prod" on a tag. Have jenkins update the tag. The tag gives you the current state, argo keeps things deployed, and you can see the deployed state with your jenkins pipelines. But the religious gitops stuff is counterproductive (in my opinion).