Thoughts on System Interfaces

I keep finding myself having the same conversation, and working on the same problems. However, today I had a kind of revelation that I think clears up a fundamental problem with in the "systems/ops/devops/reliability/etc" space that I keep finding myself in. It turns out, it's all about the api.

Now, API isn't actually the right word here, since it's not necessarily a programming interface at all. The kind of thing I am talking about could go all the way to "how one team sends another team an email to get them to do something". However, the fact that they're not automated doesn't mean they're not there. They are, constantly, in our day to day jobs as what I like to call Systems engineers. Let me be precise: in this conversation I would define a system as the collection of independent things that all work together to provide business value. This is the companies apps and code...and also the servers they run in, their storage, their CI/CD, monitoring, databases, message busses...whatever there is.

The fewer people you need to coordinate, the easier it is to build something. For this site, I worked alone, and probably have 20-30 hours of work into it total. I know lots of systems in businesses that require thousands of hours of work! As they get more complex they need to pull in contributions from more places- open source tools, vendor tools, cloud providers, ops teams, the three separate app teams downstairs, whatever. That is where this problem starts popping up. How do all of these things work together? (I like to describe my superpower as getting things to work together that nobody ever imagined putting together that way).

Imagine a classic data center distribution. A team runs the servers (vms or whatever). They try to set that machine up to be useful- maybe logging and monitoring and the like. Then another team that doesn't know much about the server comes along with their code. How do the two teams decide the process? It could be that both teams have ssh keys and log into the server and run their commands. Another option is that the platform team has access, so the app team sends them binaries or deploy instructions. If we're really getting fancy, there could be a deploy tool like Octopus for the app team, which is set up by that platform team.

The final outcome is that there is ONE thing, with the work output of multiple independent groups on it, and those things depend on each other to function. If they both just have free reign, they're going to break each other. If they don't, the team with the admin power will have to do some tasks for the other team and will guaranteed become the bottleneck. Even worse, its very hard for the teams to understand what each other are doing. The platform team could very easily have set up log exports- all log files go in /etc/my_log_folder and automatically get shipped off to elasticsearch. But, and this is the problem- there is no easy way for the app teams to know that! Maybe there is a note in a wiki somewhere, but this is the kind of thing that usually comes down to institutional knowledge, eventually you learn how to work with the provided tools. This is always the case by the way- not just servers! It could be your Jenkins installation, or your AWS account, or your JIRA ticket queue. Multiple groups of people working on the same thing in different capacities, and trying to provide tools to each other to use.

So where was I going with this? Kubernetes. I kinda hate kubernetes actually, it just feels so unnecessarily complicated. But it offers a really powerful tool- the magic of abstraction. An app team can define what they need precisely in code- 5 instances of THIS container with THESE environment variables, available at THIS dns name. And that's great, but the power is what other teams can do. The platform team can set up a logging system that will automatically pull in the logs from all of the containers running on the cluster- without having to ever know anything about those containers. The app teams don't need to know about the logging. They don't have to share ssh access to a server at all- they each provide an interface or API for the other to interact with. Look at something like the prometheus operator. The app teams get a simple api to specify what urls they provide for metrics to get scraped from, and prometheus scrapes them. They don't have to log into the shared prometheus and have write access, the operator handles aggregating all the requests from all the apps. Ingress controls especially- you define the domain name you want to listen on, and some other system you don't really need to know about makes it happen. The platform team can provide that system- without having to know about your app.

Nothing is magic. There is still institutional knowledge. How do you know that you need certain annotations on your ingress object to work correctly with how the local one is set up? How do you know to log to stdout vs disk, or that you can add labels to control what logs get picked up? But this is still very powerful, because it gives a clearly defined interaction surface. As the app team, you put k8s manifests into your namespace. As the platform team, you build stuff to handle those manifests. While you still need documentation and discussion, this is by far the cleanest separation of concerns I have seen, really allowing the team building the infrastructure to function as a separate project team and service provider...and get out of the business of being help desk for app teams who need to figure out how to log from their app.