The Truth About Business Value Estimation
In the software world, we spend a lot of time trying to decide what to do next. Should we build feature A, or feature B? How many people should we hire? Should we fix this bug? All of these can be hard decisions. However, if you do some simple math, we can make them pretty easy. The variables we need to know are-
- How much effort it takes to do each task
- How much that effort costs
- What is the value of doing any particular task
- What resources are available
- How does time affect the answers to any of these (a deal may only be good if closed by a certain date, or there may be far fewer engineers available around Christmas)
When you think about it this way, its pretty easy. Figure out some potential choices, multiply the values together, get an expected value for each path, and you have your answers!
You may have guessed that I am being a bit sarcastic here. After all, everyone knows that estimations are hard and only rough guesses. My point actually is even more drastic then that though. I believe that estimation is impossible.
Lets Talk About Measurements
Lets use a formula- how about the volume of a cube. L x W x H = V. Now, lets imagine we have a cube that is .3 inches long, .05 inches tall, and .5 inches wide. What's the volume? Punch the numbers into a calculator and we get 0.0075 inches^3. Great! Wait- are we talking about an imaginary cube or a real one? That is only the measurement of an imaginary one. In the real world, we have to think about the precision of our measurements. Here, the concept of significant digits comes into play. If we measure something as .5 inches long, we are making no claims about the hundreths place of that measurement- our instrument is not precise enough for us to. By convention, we assume that the margin of error is the magnitude of the last significant digit, in this case .1. So .5 is really somewhere between .45 and .54. The same thing applies to the other three numbers. When multiplying them together, your result can't be any more precise than the least precise input- so we can't say .0075. We should actually report 0.0.
To see why this is, lets figure the actual range. .45 * .045 * .25 = 0.0050625 and .54 * .054 * .35 = 0.010206. Importantly, notice that the more sources of imprecision there are the wider the final range will actually be.
So, Business Value
I have had so many conversations that involve estimating business values and costs. Stuff like "Well, I feel like this one will increase sales by 10% and take 3 months to build, while this other one will take 6 months but increase sales by 25%". However, I have almost never heard that conversation getting into discussion of the margin of error, and how the more uncertain factors there are, the wider that range gets. To demonstrate, say the feature involves six projects with an estimate of 1 month each. So we can guess six months, and maybe we put a month of slack in there. In reality, those six projects are probably more like .5 - 1.5 months. Which means our estimate should be 3 - 9 months, not 6 - 7. Business value is worse. The value equation usually has number of customers, rate that they sign up, how much each is worth, acquisition costs, and competitor behavior all estimated in there. The precision of each of those is also probably pretty poor. I've been in annual goal setting sessions where people where presenting plans that, in my estimation, probably had a real margin of error of -1000% to +1000%. But, that didn't stop us from arguing about whether this specific number should be a 5 or a 6, and that one over there a 300 or maybe 350 is more realistic?
So- these kinds of estimates do have value. Thinking through what the driving variables actually are, thinking about what the magnitudes may be, and considering what the drivers of that could be- they all have value. In software estimation, the process of talking through what all the steps of a project will be has a ton of value- even if it doesn't lead to a great estimate. However...take the final result with a whole bucket of salt (or just throw it out altogether). The sources of error in your system are probably big enough to completely dominate the actual result.