Flashover

There is a famous story taken from the book Art & Fear by David Bayles and Ted Orland about a ceramics teacher who divides her class into two groups: the quantity group and the quality group.

The teacher tells the quantity group that their work will be graded for the amount of pots they produce between the first and last day of the semester; the more pots a student will make, the higher grade they will get. The quality group is given an assignment to produce just a single pot for grading. But in order for the quality group student to get the highest grade, the pot should be near perfect.

The quantity group starts working with clay from day one and continues to do so until the very last day of class. The quality group on the other hand starts with research and figuring out the characteristics and methods of producing the perfect pot.

When the day to grade the students' work finally arrives, the ceramics teacher is surprised: not only did the quantity group produce the most pots, but they also produced the highest quality pots. The idea of perfection paralyzed the quality group. Instead of working with clay, the group wasted their time on research and theorizing. Meanwhile, the quantity group kept learning from their mistakes as more and more pots came out from their kilns.

The lesson of the story is simple: producing more work leads to greater work. If you want to make the perfect pot, make hundred pots, not one.

The origins of the story is based on photographer Jerry Uelsmann's university class where he conducted a similar experiment with his students (author James Clear retells the story in its original form in his book Atomic Habits). Like pots, photos can be easily produced in hundreds or thousands during a person's lifetime. However, some art projects are fundamentally lot more laborious than pottery or photography projects.

For example, author Hank Green has noted that writing ninety-nine books before publishing your hundredth, first great book is not feasible for authors. This means that if your dream is to produce a big, labour-intensive project, you need to get creative about figuring out what are the hundred pots you need to produce before taking on a more gargantuan task. What helped Hank Green to write his first novel was to write scripts for hundreds of his online videos at Vlogbrothers.

***

This post marks Flashover's hundredth post. Since the blog's inception, there's been a new post here every week.

While it has been highly rewarding to hit publish every week, and while publishing my writing every week has been a great contract I have signed with myself to force me to develop a weekly writing habit over the last two years, there have been moments where I have wished I could have had more time to work on an individual post or produce posts that discuss given topics in more depth. Because of this, this post will be my last weekly post.

Instead of weekly posts, my plan is to use Flashover to publish more long-form content and possibly write stuff on other platforms as well (I have no intentions to reduce the amount of writing I do on a weekly basis). If you have enjoyed my weekly posts, I want to thank you for being a reader. Since I won't be publishing as regularly as before, I suggest you subscribe to my RSS feed or newsletter to get notified when the next post is out.

Take care and stay safe!

Here is a term I recently learned and that beautifully describes many of the problems and questions businesses constantly deal with: FUQ. It stands for "fundamentally unidentified question." In their book Mostly Harmless Econometrics, economists Joshua Angrist and Jörn-Steffen Pischke define FUQs as "research questions that cannot be answered by an experiment."

An example of a FUQed research question is "do children who start first grade at age 7 do better in school than children who start at age 6?": a question that has important implications in terms of education policies and parenting advice.

To answer this question, we can select a group of 7 year olds and a group of 6 year olds who both start school at the same time. To assess learning performance, we compare first grade test scores between these two groups.

The problem with this setup is that older kids tend to do better at tests because of maturity; it's likely that a 7 year old will perform better than a 6 year old simply because he or she is older. To deal with this problem, we can test the 7 year olds in first grade and the 6 year olds in second grade so that everyone gets tested at age 7. However, now the other group has spent more time in school. The test scores will tell us much more about the relationship between the number of school years and test scores than the relationship between start age and test scores.

In fact, the only way to answer this research question is to study the link between start age and different types of outcomes in adulthood (such as income). In other words, we need to have access to existing data that is able to provide an answer for us. We cannot use experiments to generate this data ourselves.

While FUQs have originated in the field of economics, they are not purely academic problems. Here are few FUQs that your company is probably dealing with:

• What questions should we ask candidates in job interviews?
• What effect will a marketing campaign have on our brand?
• Will a new incentive program improve our employee performance?

You cannot run a controlled experiment to find out the link between your interview questions and employee performance because you are not able to truly assess employee performance until years later. You can run experiments to test out the effectiveness of your marketing copy, but you can't run an experiment to see what effect marketing copy will have on your brand loyalty.

FUQs are frustrating when you want to figure out the best course of action in a given situation. However, the important thing to remember is that just because you are dealing with a FUQ and can't run an experiment to justify your decisions, it doesn't mean that the FUQ is not critical for your organization.

Additional notes

I first learned about FUQs from Tim Harford's book Adapt.

In Lean Startup, Eric Ries uses extensive automated tests as an example of the different ways you can apply the ideas of lean manufacturing in software. Rob Walling, founder of the marketing startup Drip, has mentioned that having a large suite of unit tests for Drip from early on made it possible for the bootstrapped startup to move fast and refactor features as they kept learning more about their customers' needs.

As a software consultant working with multiple different projects, I have seen firsthand how test writing is often the first thing to go out of the window when deadlines start approaching and the pressure to deliver begins to grow. I'm not perplexed by this phenomenon: Tests aren't something that create direct customer value; why write a test case that users don't see if I can instead add a new feature for them? In addition, explaining the value of automated tests to non-technical people who have never had to personally experience the pain of technical debt is a considerable challenge even to the most skilful testing evangelist.

As an educator in the startup space, Rob Walling has seen an unfortunate number of software companies started by non-technical founders who have outsourced the development work and ended up with crappy codebases with no tests. After 6 or 12 months of feature development, they find themselves in a technical hole that they can't get out of without a rewrite. They might have customers but no way to build new features for them.

We tend to treat automated tests like flossing. We agree with the software experts that writing tests is an important practice, but we drop it as soon we as we are in a hurry or we just want to shave some minutes off development time. We get some short-term benefits but in the long-term we are worse off.

Since often the argument against automated tests is that we need to move faster, it's good to remind ourselves that even people like Eric Ries (one of the biggest proponents of "fast beats slow" thinking) believes in the value of automated tests. Lean Startup can be easily seen as a school of thought that encourages developers to let go of their standards for code quality and test coverage. However, automated tests enable fast learning, not hinder it: when you trust your test suite, you are able to add new features and redo existings ones without constantly worrying about breaking something else inside your product.

Teams who write tests are teams who are fast.

This week, on two separate occasions, I was reminded once again about the value of delivering usable software early on in a project.

Developer A had an uncomfortable meeting with an internal stakeholder. The stakeholder had found out that one of the planned, non-functional requirements was not going to be fulfilled due to architectural constraints. This architectural problem was not a surprise for our developer. In fact, he had warned the same stakeholder about these constraints already last year.

What was surprising for the developer was the fact that this non-functional requirement was such an important issue for the stakeholder. You would have thought that if this requirement was truly important, the stakeholder would have said something when the developer warned him that this requirement isn't probably going to make it to the final product.

Developer A is currently worried that the whole project is going to get axed because of this unfulfilled requirement. His team might have to go back to the drawing board after months of work.

At another company, developer B had just delivered her project to the client. While the developer and client were happy about reaching the end of the project, they weren't so happy about the fact that they had done so by going over budget and missing the initial deadline.

The initial time estimate for the project was six weeks. After the first two weeks of development, the team realized they are actually building the wrong features. They had to trash all the code and start from scratch.

Imagine cutting one third of your budget without reducing the scope. Even after getting some additional help and putting in extra hours, the team couldn't save the project.

The developer and client saw the project as a case study in poor planning. However, I cannot help but see this project as a success story.

Yes, there was a big misunderstanding of end user needs that could have been avoided with more open communication. But at the same time, this misunderstanding was discovered very early on in the process.

Two weeks is a lot when your budget is six weeks. But two weeks is also the standard length of a development sprint. Imagine not noticing a critical misunderstanding sprint after sprint, month after month, until you finally let the users take your product out for a spin only to immediately come back to you with complaints and disappointment.

It's too easy to keep your stakeholders waiting to get their hands on the thing you're building. We often focus on "building the thing right" and finalizing individual features before we let anyone test out the software in an actual setting. We do this to avoid unnecessary criticism towards unfinished features and confusing our stakeholders. But at the same time, we take a risk of not discovering major misunderstandings about our requirements. Developer B and her client were able to avoid a much bigger cost overrun by giving users something after the first two weeks of development.

It is true that sometimes it's not possible to deliver usable software after two weeks. However, I think most of us can agree that often we don't question ourselves enough when we say something is not possible or feasible. If we would really want to deliver something usable within a shorter span of time, we would probably start coming up with alternative, more creative ideas for achieving that goal.

Another argument against early delivery of usable software is that it might be more efficient for teams to finish one feature at a time or focus on building a strong foundation in terms of designs, infrastructure, and background systems. However, this approach will also increase the risk of costly misunderstandings.

You take a gamble when you don't ship anything usable early on in your project. If the risk of possible misunderstandings doesn't manifest itself, you should probably consider yourself lucky – not a master at planning.

In 2010, Nokia's head of devices gave a strategy update to the company's board of directors. Nokia was falling behind the Apple and Android ecosystems in terms of delivering new features and phone models to consumers. There had been yet another release delay because of problems in the development of Nokia's new operating system Symbian^3.

One of the board members, Risto Siilasmaa, couldn't believe his ears when he heard about the issue of Symbian's compilation times: it took 48 hours to compile Symbian (compilation is the process of turning your code into something that machines can understand). Compilation times are normally measured in seconds or minutes. Large systems can take hours to compile but a compilation time of two days is beyond anything reasonable.

Unfortunately, this was only the tip of the iceberg. Siilasmaa later learned that the overall build time of Symbian was two weeks. This meant that it took developers two weeks to see their code in production after they sent it off and moved on to the next task.

Symbian is a cautionary tale of insurmountable technical debt (also known as development debt). But what is technical debt and how do we end up accruing it?

Because development teams have to meet deadlines or work with limited budgets, they sometimes end up taking shortcuts such as duplicating code, hard-coding a value somewhere, or not writing automated tests for new functionality. These kind of easy fixes often impact future productivity because they tend to make future code changes more difficult or laborious. For example, a missing test case usually means that developers have to do more manual testing every time they push new changes to the codebase.

Avoiding technical debt at all cost is not necessarily the ideal. Also, there isn't really a scientific way to calculate the perfect amount of technical dept. Development teams have to constantly find the balance between budget, technical debt, and future requirements. Avoid all technical debt and you might not release new features fast enough. Take too much technical debt and your feature development might come to a standstill because all code changes suddenly take months to complete.

The likely reason why Siilasmaa was the only one in Nokia's board of directors shocked by the compilation times was that Siilasmaa was also the only board member who had experience from software. Before joining Nokia's board, he had founded and managed the publicly listed software company F-Secure. This is Siilasmaa's advice for companies who find themselves in a hole of technical debt:

My own experience with development debt at F-Secure had caused me to conclude that there's only one solution: As soon as you realize that you have fallen into this hole, create a separate team to start a complete rewrite of the source code or do the rewrite in stages. Assign separate resources for reducing the debt: otherwise it will never get done. Do this even if it means recruiting a new team or making a bigger investment than anticipated. The sooner you start, the sooner it's ready.

At some point of their history, tech giants such as Microsoft and Amazon have had to implement a feature freeze because of gargantuan amounts of technical debt that was impossible to untangle without a rewrite. These feature freezes are not cheap. For example, Amazon's rearchitecturing project took them six years and cost over $1 billion.

While introducing new technical debt is not a mistake, not acknowledging the fact that shortcuts and easy fixes come with a price is.

Final notes

I first learned about Symbian's compilation time from Gene Kim's business novel The Unicorn Project. Risto Siilasmaa's book Transforming NOKIA can be found here. Both books are great reads for anyone looking for great storytelling in the tech space.