Paradox of the IT Cost-Benefit Analysis

Inside your organization there's an initiative for a new software project. You are asked to prepare a cost-benefit analysis for it. Your organization needs to be sure that it's allocating its limited resources for the right projects. People calling the shots (decision makers and managers) want to see a calculated return on investment. While they have appreciation for the qualitative aspects of different initiatives, it's hard for them to compare different projects without valid quantitative measures.

How will you start? What will you turn your attention towards? Which part of your analysis will be the most sound?

It's very likely that the bulk of your effort will go towards estimating the development costs. It's what most of us focus on. It's what most of our managers and clients want to know. But cost analysis is not the most valuable information for decision making.

Douglas Hubbard (author and consultant in decision sciences) uses the term "IT measurement inversion" to describe this paradox. According to Hubbard, "most organizations spend a lot of time estimating initial development costs but not so much estimating benefits, how fast people will start using the system or the chance the project will never be finished." However, the probability of project cancellation and the utilization of a system being developed are the most important unknowns organizations should try to research.

This doesn't mean that you shouldn't estimate development costs. What it means is that you should instead rethink how you prioritize different variables in your cost-benefit analysis. The idea is to allocate the majority of your research time for figuring out whether the project will get terminated or not. Second biggest chunk of your time should go towards estimating system usage and how fast that usage will grow (i.e. how quickly you'll get people to use the system).

Information has value which is calculated by multiplying the chance of being wrong with the cost of being wrong. If you decide to invest into a software project but the project fails, your cost of being wrong are the development costs. If you decide not to invest, your cost of being wrong are the missed benefits.

The mathematics of information value explain why project cancellation is such a critical unknown. When a project is cancelled, you miss all the benefits and incur considerable costs. A chilling example of a terminated project is LIDL's decision to cancel its switch to SAP after spending seven years and €500 million on it.

System usage is the second most important unknown because benefits are generally assumed to be larger than costs. If the costs were initially assumed to be larger than the benefits, the project wouldn't even be considered as a viable initiative inside your organization.

Remember the formula for information value: chance of being wrong × cost of being wrong. When we are able to decrease the chance of being wrong about the likelihood of project termination and benefits, we achieve greater gains compared to decreasing the chance of being wrong about the development costs. When it comes to development costs, the cost of being wrong is outweighed by the cost of being wrong about possible benefits and project cancellation.

We automatically direct our focus on development costs because our minds are trained for that. First of all, we, as humans, suffer from loss aversion (we prefer to avoid losing $5 than finding $5), and at the end of the day, organizations and committees consist of people like us who carry different cognitive biases.

Secondly, since we are so often asked about costs in terms of money and time (e.g. story points), it's easy to start assuming that costs are somehow more important than benefits. However, it's very likely that our organizations are already giving enough attention to cost calculations. It's time to start developing our skills in calculating the benefits.