Infrastructure as Code

I've seen and created infra-as-code setups at quite a few companies at this point, and seen/talked about a ton others. At this point I feel like I have a good idea of some practices that make things work better, and thought it may be helpful to try and enumerate them. What values make an IAC setup good, and what practices can help us get there?

Declarative config is better than Imperative config, even if declaring an imperative series of actions!

The truth is that imperative infra is one of those things that seems simple enough to script (do A, then do B), but it can be really hard to do effectively. Sure, create a new VPC, then create a routing rule, then create a VM. Problem is next time you run it. The first run, you go from known state (the thing doesn't exist) to known state (now it does). Next time though, you are going from unknown state to known state. You have to handle all of the cases of that VPC (does it exist? Is it named properly? Are all the attributes set correctly? What if one of those attributes is wrong but requires deleting and recreating, do you handle that?

In the end your script will either only handle the happy path, or get very complicated quickly. So, if at ALL possible, use something like Terraform that splits the logic of going from state A to B out from the logic of declaring the desired state. If you have to write the code yourself, still take that approach- separate out what you want from how you get there.

Every chunk of infrastructure code can be built up of three parts- the resource provider, the declarative description, and the parameters

Hopefully you don't have to write the provider- that's the one that says "given that we want a VM instance, this is how we get the current state of that instance and mutate it to the desired state". I chose the word provider because that's what Terraform calls it. Ansible uses 'module', Kubernetes uses 'controller'. If you do hit a case where the thing doesn't exist though, even if writing your own custom code, it still makes sense to think in those terms. Have a chunk of code that accepts a declarative description, can determine the current state, and can handle mutations- separate from the description of your actual infra.

The description would be a terraform module or ansible playbook or helm chart. "Our EKS setup consists of a VPC plus 7 security groups plus...". This is usually the place that an infra team will spend their time iterating and working to get stuff working. In 99% of cases, you will need to deploy more than one copy of whatever this is, which is why we use the term module. It is usually not worth trying to be too general! Unless you're writing code for broader consumption (you're a vendor selling an on-prem app or an open source maintainer), you are not going to need a ton of variability- build it as you need it.

The parameters are the variables. "Environment A needs 6 VMs, but Environment B needs 10". "We want to deploy a copy of module 3 to QA". This should be your next level interface between the infra experts and the people who want to use the infra. If someone needs to know about what is deployed, they would look at the parameters. The parameters MUST be stored somewhere as state- if people are running deploys directly by passing variables on a command line, you have problems- nobody will feel confident that the actual state matches the desired state, or feel safe making changes.

The three sets of code should be stored in separate places

Ideally, you are consuming providers from a third party of course- but if not, it is still important to separate it from any modules using them. The pain of splitting the modules and providers if you later need to use the provider again for some other application will be far greater than the extra friction of starting off split. It could just be folders in a repo, but I think it is more effective to use separate repos. This lets you easily have different modules using older versions of the provider and to iterate on them separately. The exception would be if your provider describes some resource and it would only change if the underlying service changes- in which case keeping them in a repo so everything changes at once probably makes sense.

The parameters definitely need to be separate from the modules. I am not convinced that using git for this even makes sense (https://paulbecotte.com/blog/post/-gitops-is-an-anti-pattern) but we don't really have better tools for the purpose. A separate repo with nothing more than the desired parameters works well. Each parameter file should refer to the module it is instantiating through a git URL, allowing you to version the module. You must do this so that you can do deploy pipelines. Update staging, check that it works, THEN update prod. The limitations of git here though mean you have to do some pain to get this workflow. The best approach I have seen is that the deploy pipeline for each module will run a script to update the version reference in the config git repo. So the pipeline will be "test, checkout config, update staging.json, commit, wait for tests, update prod.json, commit". There's probably a good opportunity for a tool here, and I am still open to better ideas (a real database seems to make sense lol). I usually wind up with one config repo per team/org, with lots of module repos.

All apply operations should be run by your CI system

If you're typing `terraform apply` at the command line in the normal course of business, your system is broken. This is because nobody (including you!!) knows whether the state in the config repo actually matches what is deployed. In a team of people, you'll find yourself asking each other if the repo is up to date and being afraid to make changes. I get it- when iterating you'll probably want to apply to a dev environment directly, but if at all possible you should remove the ability for yourself to even be tempted to make changes to staging/prod. I had a job where we put off CI/CD of our IAC pipelines. Our team moved slowly and accomplished little. It is a decision that I regret and won't make again. If you want IAC you have to ensure that the code is truly the only source of truth for the deployed systems.