You Keep Being over Budget Without Data
Software developers are constantly being asked to estimate work. The work to be estimated can be a small task that requires input from an individual developer or a feature that involves a full team of people. Estimations might happen during a weekly meeting or while drafting a project proposal.
Software developers are also constantly wrong with their estimates. We keep thinking that things are easier than they truly are. We all go over budget and miss deadlines all the time. On Monday, you started working on a feature that you thought was going to be out by the end of the day, and on Friday, you were barely able to submit your pull request before heading home for the weekend.
This repeating pattern of underestimating the difficulty of future tasks is referred to as the planning fallacy and it's a result of multiple factors: We present overconfident estimations because overconfidence is often rewarded in our society starting from early childhood. We omit the time required for coordinating work from our estimates because our brains are not good at understanding how difficult it is to build things together with other people. We procrastinate because we have a tendency to go for the tasks that give us instant gratification.
Famous psychologists Daniel Kahneman and Amos Tversky argue that since humans are inherently bad at estimations we should rely more on data driven planning. The first step towards data driven planning is to start tracking performance. Unfortunately, we can't trust ourselves or others to provide us accurate historical data about past projects. Our brains are not always able to think mathematically or store information in a reliable manner. Therefore, we need to collect our plans and the actual durations and costs to some kind of database.
Researcher Yael Grushka-Cockayne's ideal database of completed projects includes the planned schedule, information about project tasks, and actual durations and costs. She's also suggesting that in the future we will be applying machine learning for our project plans. Here's a link to her 10 minute talk about the planning fallacy and data driven planning.
Even after you have collected a comprehensive database of past projects, your estimations aren't going to be accurate all the time. But this is not a good enough excuse to ignore tracking. Estimates based on historical data are still going to help you miss deadlines and spending limits less often. In turn, this will lead to better consistency, less stress, and more trust.
It's also true that our work as software developers is creative work characterized by many unknowns and novel domains; no two projects are alike. However, you can still identify reoccurring problems within your work. We end up building authentication systems, retrieving data from external APIs, and designing CRUD features over and over again. You can break your projects into separate chunks that you are able to estimate using historical actuals—but only after you have started to collect the performance data required for those estimates.
You are probably able to recall examples inside your company where projects ran over budget. When it comes to your own work, you know that you keep being overly optimistic about available time and required effort. Is this the time for your company to start tracking plans and actuals? How about yourself? Should you start tracking your personal estimates and actual durations?