The Outrageous Cost of Skipping TDD & Code Reviews

TDD is the process of writing automated tests to ensure that code works before writing the implementation. You write a test, watch it fail (red), write the implementation, watch the test pass (green), and refactor if needed. Repeat the cycle as you build out the system.

The process has been studied in depth and has proven itself to be very useful to increase the quality of software. But did you know that it also saves organizations a lot of time and money?

One of the primary reasons managers cite for waiting so long to implement TDD is the cost. It’s common for initial project build-outs to take up to 30% longer with TDD.

What those managers need to know is that TDD reduces production bug density 40% — 80%, and that makes all the difference. More bugs in production lead to a dramatic rise in maintenance costs.

According to the IBM System Sciences Institute, fixing a production bug costs 100x more than fixing a bug at design time, and over 15x more than fixing a bug at implementation time.

 

Using these figures, let’s look at the relative costs of a fictional project with and without TDD. The project will take 1,000 hours of up-front implementation without TDD, not counting maintenance:

In our fictional example, we saved 623 man hours, or, for a team of 4, about a month of development. If each developer is paid on our yearly average ($30k) and factored an average of 30% on top of developer's cost to cover benefits, we reach a savings of almost $10,000 USD.

Of course, this is a fictional example making a lot of assumptions. Your mileage will vary based on lots of factors, including your team’s experience with TDD, the presence or absence of other quality measures (such as code review), etc…

Here are the assumptions made:

• Production bugs without TDD: 100
• Production bugs with TDD: 40 (60% fewer, middle ground between 40% — 80%)
• Average time to fix bug at implementation (TDD) phase: 1 hour (this number is only used to derive the production bug fix cost)
• Average time to fix a production bug: ~15 hours

Playing with any of those variables will obviously alter the results. What is pretty certain is that TDD saves time & money as maintenance costs get factored in — a lot of time & money.

The relative results of this fictional example ring true to me based on my experience with real production projects, both with and without TDD.

Code Reviews Have Similar Effects

Code reviews have similar effects. In fact, some studies have found code reviews are more effective than TDD. According to a 1988 study, each hour spent in code review saves 33 hours in maintenance.

That’s pretty extraordinary, but the bug catching benefits of code reviews just don’t live with my personal experience. I find automated tests far more useful at catching bugs — so how do we explain the extraordinary effectiveness of code reviews?

Knowledge sharing. When you employ code reviews, your team self-corrects when developers are using anti-patterns and other poor practices. At the same time, they’re sharing effective patterns, teaching each other better ways to do things, and teaching each other how to write more readable code.

The net result of that is that your junior developers quickly rise to the level of the smartest developers on the team, and your whole team velocity improves. It’s hard to over-state the value of a mentorship culture on your team.

The Cost of Interruptions

Why does it cost so much more to fix production bugs? Because once a bug reaches production, the cost is much more than the simple cost of fixing the bug.

Fixing a bug in production is much more expensive than fixing a bug during development. To understand the magnitude of the difference, you must first understand the cost of interrupting a developer.

A production bug fix frequently means an interruption in the context and cadence of feature development. In other words, developers are pulled out of the context in which they’re currently working and dumped into the context of the bug, where it takes a time to absorb the related code and flow, diagnose a root cause, fix the bug, and then reabsorb the context of what they were working on before.

Each context switch can cost up to 20 minutes of developer productivity, but the bleeding doesn’t stop there. Interrupting a developer to fix a bug is likely to cause more bugs in the code they were working on before they were interrupted. According to Microsoft Research, an interrupted task takes about twice as long to complete and contains twice as many errors as an uninterrupted task.

In other words, the cost of fixing bugs that get released in production isn’t just about the cost of fixing the production bug. Interruptions increase the cost of current development work and introduce more bugs that will eventually need fixing, too.

On a related note, multitasking is a really bad idea. To be productive, developers need to focus on one thing at a time.

Conclusion

The next time somebody tells you they don’t have the time or budget for TDD or code reviews, tell them with confidence, “in that case, you really don’t have the time or budget to skip TDD”.