Making Devops Work

I had the following exchange on Twitter the other day-

Which made me want to respond, but not in a 140 character way. You see, this is something I feel really strongly about. Not about source control not being important or anything like that- it is about the term itself, DevOps.

I am not sure why sysadmins started getting called "DevOps" engineers. I have even been hired, specifically, to be a DevOps engineer. Now, I will admit that I am pretty good at the things that people think of when they say "DevOps Engineer". I can automate, and wrangle Jenkins, and know the ins and outs of several source control systems (Hi Clearcase!), and have written really nifty deployment scripts and can speak several configuration control system languages etc. All of this is great (and a worthy investment, in my opinion). But it isn't really DevOps. It falls under the heading of Continuous Delivery, yes, but not DevOps.

Even better is the advanced stuff that a really good "DevOps Engineer" knows how to do these days. Things like advanced monitoring systems, resilient and self-healing architectures, infra stuff like Kubernetes. These are all great things to have in an advanced system- they will make it work better, reduce outages, and shorten the feedback loop that is so important to DevOps, but they aren't it. The best term I have seen for this stuff is "Site Reliability Engineering". Practices for how to engineer a distributed system to be more reliable and resilient. Great stuff.

This may seem a bit pedantic, after all, arguing over the meaning of buzzwords is kinda old on the internet. But my problem is that they miss the point. The whole purpose of the word DevOps is right there in the phrase- Dev and Ops. DevOps shouldn't be a fancier way of doing operations work, it should be a better way of running an organization that's primary goal is delivering software.

Many of us have been there, and many of us still are there. Where developers feel like they can't get any work done because the operations people are a giant roadblock, and the ops people constantly feel like they have to protect the world from the dev guys who don't know anything about how applications actually get deployed, and have a work queue three times as long as the people they have to work on it. This is the inevitable result of splitting the two things into separate jobs with competing goals. One's job is to change things as fast as possible, while the other is to prevent change.

DevOps says that the best way to overcome this isn't by having your ops guys set up a Jenkins server... it is by having your ops guys and your dev guys work as one team to build and deploy and service one system! Pretending that monitoring and security and deployment are some separate set of things that get patched onto software when it is done is foolish and short-sighted. Instead, work as a single team. The work of building in security and adding deployment scripts to a project are equally important to be in the project backlog as adding feature 43 or widget B.

I have found that by having the work be part of the same queue, and having everyone on the team do parts of it, a couple things happen.

• Deployment, monitoring, security, etc get built into the product in a much more fundamental way. The ops people aren't trying to monitor an application they know nothing about from the outside... they can build it right into the app.
• Knowledge gets distributed. Devs knowing how the platform works, and Ops knowing how features work leads to a lot lower bus factor (and a lot less time with the two groups going back after the fact to explain that "yes that is the way the deployment process works").
• Features on both sides of the divide get built faster- instead of waiting for an Ops project to get done, if it is the most important thing, extra people can help out.

Now, I guarantee there are sysadmins and SREs and even devs reading this and thinking this is a call for "no-ops". Its not! I happen to believe very strongly that having people on your team with expertise in all those technologies I mentioned above is a very valuable thing. However, I see it as valuable in the same way as you might have an expert in Swing and an expert in Angular in a particular application using those technologies. Having different expertise does not, in my opinion, mean they should work from different queues and throw tasks at one another- it means that the team has different people to ask for help and lead up different tricky parts of the system.

This even extends to supporting the thing. Everyone on the team should spend time dealing with bug reports and outages- that knowledge should influence every line of code that is written, not just deployment scripts! Of course, it means that people hired for this probably need a more diverse skill set- a sysadmin with 0 programming abilities probably will not flourish under these circumstances.

Ultimately, the more closely I have seen this followed, the more rapid and smooth software development seems to go. Working in this way- as a single team working towards a single goal- leads to faster iteration, greater reliability, and greater job satisfaction for the team members. So, that is why I choose to argue about the meaning of the word. Breaking DevOps into "Jenkins expert" means ignoring the team structural changes that bring the real benefits. Sure, practicing Continuous Delivery is really useful, and source control is essential to get there. But even more essential is structuring your team and workflow in a cross-functional way... since that is the shortest path to actually delivering the "Continuous is a lot more often than you think" promise!