Deciding How to Work

I've been on enough software teams by now, in enough different organizations, to have seen lots of different ways of running projects. We've all seen the usual "Scrum" and "Kanban" styles (I put them in quotes because its pretty rare for any team to run either of them the way they are described lol), have even had a manager who really liked to use excel and multi-hour "standup" meetings. I have participated in numerous super contentious discussions about the specifics of how a team is going to manage its work, but looking back, rarely about why they would prefer one over the other.

I think if we start from first principals, we may be able to get better consensus and agreement on how we should manage our projects. Let me list some observations / opinions -

• It is important to limit context switches as much as possible to allow more predictable delivery

• Unplanned work destroys consistency, but is a reality- any system of working that ignores it is not serious

• Prioritizations change all the time and it is important to focus on current priorities, but changing too often can make it impossible to ever deliver anything large / complicated

• Left to themselves, engineers will mostly prefer to work on their own things by themselves (avoiding coordination costs). However, the total knowledge of the system and overall team performance benefit from more collaboration.

• Any part of a project that is mostly laying groundwork to later deliver user features will take much longer than expected and involve fewer people, with the end result that the actual features never get built because the first part uses the whole schedule

• Parts of a codebase that are hard to test or deploy will only become more so over time, and will eventually grind overall development to a halt as people focus on changes in the other parts of the app

• Everyone will usually agree that documentation and refactoring are valuable, but will usually not prioritize working on them.

• Teams will usually have topics they disagree on - "Should we change the whozit API to accept foo objects?", "Should we deploy with docker containers?" that just get discussed constantly, burning time without ever actually coming to a consensus (and as such, the work not getting done either way).

(As an aside, the last point is probably my greatest personal failing- it takes an active effort to force myself to leave the bike shed alone!)

For a team that these issues sound familiar to, we can start to think about ways we could want to improve our workflow to help with these problems. Every team is different, and the actual solutions may vary, but let us consider some of them and possible effects.

Assign a person or persons as "On Call"

For a period of time (during work hours) they are responsible for being the primary person handling unplanned work. This is a tantalizing idea, especially if a team has many manual processes that have to be routinely handled. I imagine this can do a lot to help with unplanned work and context switching, but has downsides. One of them is that project work that depends on a specific person could now wind up getting blocked for a day or week or whatever the rotation is. A team that used this workflow that I was involved with had this issue often, where tasks would get held up. It also didn't work as well as hoped for the purpose of reducing interruptions. It worked well for the manual processes, but our team was very silo'd and most team members couldn't handle most "problem" issues without involving someone else who was presumably doing project work. A different team that was more cross-functional though, it worked a lot better with only major issues disrupting the rest of the team. That org had fewer manual processes though, and had the opposite problem there, where manual processes (creating employee logins in our database was manual for example) wound up getting owned by specific people.

Overall, I think this is a good approach so long as you take great care to work on knowledge share of team members so that the on-call person can deal with most issues and project work can be shuffled around as people go on the schedule. An alternative could be doing a one day rotation instead of a week like I had in both cases, so that tickets could just sit for a day or similar. I also believe it is incredibly important to track the number of unplanned work items in a ticketing system so that you can focus on bringing the number down over time. If forced to choose one metric that is a leading indicator of software delivery success, I would pick "percentage of planned work vs unplanned".

Work in Process Limits

Kanban is usually interpreted in software teams as "have a backlog of tickets to work on", when in reality the WIP limits (which teams tend to skip) are the defining characteristic. Having a WIP limit on developers can have some effects that I think are positive. The biggest one is on predictability and context switching. The default when an engineer gets blocked on something (they are waiting on ops to open a firewall port, or a long deploy pipeline to run, or an email answered by a customer) is for them to start something else. My was on a team that had 7 people on it, and it was extremely common to have ~50 tickets marked as "in progress" at any given time. The observed behavior in that case is that nobody has any idea if any of those 50 tickets were ever actually going to get done, and certainly not when. We wound up with tons of projects half finished, where someone started working on it while blocked on something, then got called to something else and it was never the most important next thing. This is a pattern I have seen everywhere, though probably not so pronounced. We have all been taught that we should be efficient and always working on something. At Narrativ, when the team was most effective, we tried enforcing a 2 ticket per engineer WIP limit. It seemed to show good results, but had an unexpected effect. Engineers pushed back really hard! I told the team that if we are at the limit, don't start a ticket. Do QA/PR reviews, ask around and see if you can pair with someone else, or even do nothing at all. I would have expected having the boss say "just do nothing" would be something people enjoyed, but it was very much the opposite. As of today, I believe that the positive effects can be had this way. We can decrease context switching, have people blocked less often (more people available for stuff like code reviews), improve delivery predictability, and improve collaboration- but it is the single rule I have tried that required the most enforcement and buy-in.

Reserve Capacity for The Team

There are multiple items on our problem list related to quality. Unplanned work and hard-to-change code come around from maintenance work that has been delayed too much. It can be very challenging to prioritize "improve the deployment script" versus a user feature. However, that work is a force multiplier- doing it helps the team deliver features faster over time, and if its not kept up on, will eventually destroy the team. As such, I don't think its worth even trying to do so. Reserve a percentage of your input queue (tickets selected for work) for workflow improvement tasks. I like 1/5, but adjust based on taste (and size, user stories are often much less granular than the team tickets). In something like JIRA use a different issue type, so you can tell at a glance how much of your queue are these kinds of work items.

(As an aside, the Phoenix project describes four types of work- projects, internal projects, routine changes, and unplanned work. In JIRA or your ticket system of choice, using different issue types for these four feels good. Your goal is to understand the routine changes, keep the ratio of projects to internal projects relatively fixed, and decrease the percentage of unplanned work tickets over time)

I have never seen a team that didn't do this keep the code base safe to change in the long run. It goes along with the previous point, these kinds of tickets can feel the same as telling people not to do anything. Everyone wants to be efficient. In both cases though, NOT working on the prioritized business projects in order to work on the workflow itself can help to keep the team efficient over time.

Force People Out of their Comfort Zones

This is probably the most important. The usual case is that the person who writes a chunk of code will be the person who continues to work on that code. The deployment system needs a change? Oh, that's Ken's ticket. This results in every "kind" of work having an invisible queue based on the specific person who knows how to do it, along with the fact that each person eventually becomes a point of failure. Each time you finish a feature, you increase the number of things you're responsible for, until you get to the point where 100% of your time is spent on supporting things you built earlier and you can no longer do new work. That's the point where people find a new job. It also means that prioritization discussions can be tricky. You pick a ticket related to the database because Tina doesn't have anything to work on this week and she wrote the ORM.

Instead, force people to work on tickets that they do NOT already know a ton about. This spreads knowledge around the team, makes it so that different prioritization flows can work, allows more people to handle unplanned work without interrupting each other, and allows the team as a whole to continue to grow and develop new features. This gets tons of pushback when you try to implement it, but was probably the one practice at Narrativ that was most responsible for helping the dev team be more productive. Especially if you start to develop a hotspot- one expert who handles all the tricky parts, its very important. Set it up so that issues cannot go directly to that person- someone new will handle them. If they don't know, they can ask the expert for help, but the expert can only advise, they cannot take over. Eventually knowledge will spread around to the rest of the team.

Don't Have Meetings Without an Agenda, But Make Decisions When You Do Have Them

Engineers can debate tabs versus spaces endlessly. Or even worse, never debate and have contests in the codebase to change things to their preferred style. Then someone will call a meeting to talk about the decision/design, talk past each other for a while, and then say we need another meeting. I would argue that in most cases, making a decision- any decision- is better than making a perfect one. If you are going to have a meeting about a design / style / proposal, it must be structured. Lay out the specific question(s) that need to be answered, enumerate the possible choices, and have a person(s) argue the advantages of each one. When the meeting time comes close to being over, go ahead and pick one- never leave it unresolved. Then all agree to stick to that decision and not argue about it every week. For me personally, this is the point that leads to the most improvement- but arguing over tech choices is a vice almost all of us have.

In Conclusion

This is just trying to lay out some of the first principals we want to accomplish with a project management system, and some practices that I think can contribute to them. A common pattern seems to be that everyone in the engineering org is running around in circles dealing with firefighting, while doing some process like scrum for presentational purposes. I would argue that if building a sprint plan based on a couple hours of work that there is zero percent chance of hitting isn't actually helping you, there is no value in doing it. Similarly, if you don't find value in WIP limits on a kanban or whatever other practice, then you shouldn't do it! Instead, decide what you want to accomplish. What pain points prevent your team from delivering value in a safe, sane, and predictable way...and then choose practices that you think will address those specific pain points. Then, after trying them out, check back and see if they worked. Continuous process improvement is the holy grail, but can be very hard to do right. Us engineers love to lie to ourselves and follow cargo cults, but we rarely stop trying to flag down the cargo planes even if they don't show up.