There are a lot of different buzzwords to describe the people and jobs that take code and build/deploy/run a software system. Devops, SRE, Platform Engineering, Ops, Sysadmin, Release Engineering, probably a bunch of others. I recently took a job as "Lead SRE", where the team is currently primarily involved in the tasks "take the code dev teams have written, and deploy it". I titled this "Release Engineering" because it feels like the most accurate name for what we are doing today, but different companies call this set of tasks all of those things. As I work with the team to understand "how can our team deliver more value to our business" I thought it would be helpful to consider what the business oriented goals of this task are (versus the engineer centric ones like "Jenkins sucks").

Top Level Goal

So- what is our prime directive? Tech companies deliver value by writing code and delivering it to production. The Devops Research and Assesment Group specifies two metrics to track how fast we deliver that code, "Lead Time" and "Deployment Frequency". These are very much oriented around minimizing the gap between "code written" and "delivering value to customers, and discovering if the new code actually accomplishes what we hoped so that we can feed that information back into the development process". This seems like it'd be easy- just throw every code change forward! That said, there are a couple competing goals.

• How often does trying to change the system wind up breaking the system instead (This is the metric** "Deployment Success Rate"**)
• How long does it take to fix the system when something goes wrong ("Mean Time To Recover")
• How much effort does it take to release the change (since that effort is not going into working on the code)

These are important. The system my team has today involves quite a bit of effort from many teams to do any deployment of new code, and the system is complicated- errors or downtime can cost customers money but its very hard for everyone involved to understand how any particular change is going to work or not work. These are strong incentives to reduce velocity, even though we have agreed that velocity should be the key goal for the team.

What Properties of the Release System Support the Goal?

Now that we know what we are trying to accomplish, the question should be - what things do we have to optimize for to support moving those metrics in the right direction. In my opinion, Psychological Safety is the most important thing a system can provide to move all of those metrics in the right direction. If engineers feel safe that they can change any part of the code without having to worry about the fifty other things they may have broken, they will be able to make changes more quickly and confidently. The opposite is often true- there are parts of the system that nobody really understands and they just avoid ever messing with it. The infrastructure especially is often in that camp! So many different teams usually contribute to it. Something like "Can I shut down this VM?" are the kind of questions that people are scared of. This means that we don't do things like "upgrade the JVM" because we don't feel confident that we have a way of knowing the full impact of that change before we try to deploy it.

Visibility plays into that. Even if you do feel safe to make a change, and more so when you don't, you need a good way to know whether the change worked. Did you break something? Is the system doing what you hoped? Is anyone even using the new feature? In order to move quickly and safely, everyone in the org needs to be able to see as much data as possible about what the system is doing. It lets you look for unexpected changes, analyze the best way to approach any particular problem, and know more confidently what is happening. Its also very important that it is very easy for everyone on the team to set up visibility on new aspects. So many teams are just looking at monitoring as CPU usage. Custom metrics that track custom app specific things are the holy grail here. I have seen some of the best value in my career when we just added a special log message to the code when something happened, and setup a dashboard to tell us how often that log was emitted.

Just as important as knowing what is currently happening, is knowing what is going to happen. Production Like Environments (that are not prod) are the key to this. In order to feel safe you have to know what a change will do before making it, and in order to do that you need confidence that you can try out your change before making it. Having the system provide easy and straightforward ways to try out changes in a way that is very much like production is key. Interestingly, its probably most important for the actual change system! If the only time you ever go from version A to version B is when you run the deploy to prod, you are going to find out all sorts of things that didn't work as expected- and then have the fun experience of fixing it live.

So, What Should We Build?

I don't want to talk about specific features, but rather the approaches. The goal is to deliver business value by moving those key metrics in the right direction, which happens when our features support the three properties.

• We need the full deploy to be scripted to a single command. Every time a worker is expected to do things in a particular order, there's a chance they do not do so, or misunderstand the logic, or hit the wrong button. This means that "deploy checklists" are just not an acceptable approach to this problem, because there's no way to make/implement one that moves these metrics in the right direction. Encode all the deploy logic in code, test the code, and run it.
• We need deployments to be idempotent. The logic of whether to run a step/change a thing should be encoded in the system, not in the people running the system. This is MOST important for things like database migrations! We need to be able to run the script all the time without worrying about whether its going to do the right thing.
• We need the environment configuration to be captured in the code. An environment where three or more teams are deploying things to an environment that all depend on each other is very far from optimal. Worse, it freezes all of those teams. You can't feel safe making a change if you have no idea who else may depend on your stuff. Therefore, everything that the app and deploy depend on should be captured in the deploy system, even if its expected that thing may already be setup by some other team.
• We need comprehensive observability. This includes, at a minimum up/down health metrics for every process, plus events like "deploy" and "migrations". This is how we gather the metrics listed above, and gathering them is the first step to moving them in the right direction.

Conclusion

As the team responsible for releasing an application, we have an outsized impact on a lot of the things that can make the overall organization more successful. We can make it easier and safer for developers to deliver their code more rapidly and confidently to production, and make it easier to measure whether that code is having its desired impact. We are also likely the experts in the overall system and how all the pieces fit together. As such, we have to optimize our systems to deliver the key values listed above, with the goal of moving metrics like lead time and MTTR in the positive direction. The best way to do that is to optimize the deploy system by making it a single command that can run without a ton of external setup, and will do the right thing no matter the current state of the system. Then we exercise that system all the time to increase confidence and reliability while enhancing observability so that we can prove things are improving.