BOOK CLUB: How to Reduce the Cost of Software Testing (6/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. Today's entry deals with Chapter 5.

Chapter 5: Trading Money For Time: When Saving Money Doesn't (And When It Does) by Michael Larsen

I vacillated as to whether or not I should do a book club entry on my own chapter, but since I already have chopped up the review into 21 parts, and my chapter is part of the book, and it is written in context of this first section, I figure I might as well include it :).

During economically challenging times, many organizations opt to use the tools already in place (if any exist) and machines that are already at work in their current state. Money is not spent for extra infrastructure, or extra people, so costs are lower. What this “cost saving” doesn’t address is the time trade‑off made to perform these tasks. The Cost of Testing is not exclusively about money, very often we must include time in that cost analysis as well.

We've often heard that “You can have a system that is cheap, fast, or of good quality. Pick two.”  If one area gets incremented, others are decremented. Spend less money to perform a task, the time to accomplish the task often increases. The quality of the resulting output may also decrease. In my experience, companies want to bring costs down, while keeping the quality level as high as possible. To make both goals succeed (lowering costs and improving quality), in most cases we will have to allow time to increase.

Some people recommend that our full 24 hour day should be viewed in light of what we earn. To really determine the value of our time, we would need to take the total amount of time in a week, and divide it into an hourly figure. If we were to divide those hours in comparison to an average software developer earning $50/hour (an arbitrary figure), that individual is really earning $11.90/hour. This is the concept of the "real wage", and it was popularized in the book "Your Money or your Life", written by Vicki Robin, Joe Dominguez, and Monique Tilford. We can also look at a less extreme example. When all activities and expenses that immediately impact our work are considered, the difference in the real wage can be anywhere from 30%‑50% lower than our earned hourly rate. The point is, when we realize what our real wages are, we feel prompted to do significantly different things with our spending and behavior (and our time).

The value of time becomes more critical as additional people come into the picture. Time is multiplied, and each non‑optimized hour affects all of the individuals involved in a project. The "real wage" for an organization can be greatly increased by its ability to save time. It can also be greatly decreased by the time it loses.

When I look to balance and consider the costs associated with money and time, it’s important to look at both short term and long term views. There is always a return on investment when it comes to the time put into any process vs. the value of the output. Whether the return is positive or negative, and by how much, is up to the team and the organization. Finding the magic point where we can save money and keep to a good amount of time while ensuring high quality is a delicate balance. Having all three perfectly balanced is unlikely. There is a way to get to where the monetary cost is “low enough”, the overall time is “long enough or short enough” and the quality is “good enough” to be effective and release a good product to stakeholders who want it in a timely manner.

Determine the Most Critical Areas to the Stakeholders

One of my development manager asked our assembled group: “Do we make software here?!” When the developers and testers said yes, he shot back “No, we do not! We make a solution that solves a problem for our customers. At the end of the day, if we don’t get that part right, it doesn’t matter what else we did get right!”

Focus on the Highest Risk Areas First

The areas that matter most to our stakeholders are ultimately the highest risk areas. It doesn’t matter if we agree or not, they are the ones who want the application, and their wish list is what’s driving the purchase of the application.

Make Sure to Ask the Right Questions

Who are the intended users? Does a product already exist that meets this need? If we are first to market, does the application do what our customers want it to do? If we are not first to market, what differentiates us from the competition? Is that difference significant enough to make for an appealing alternative with the established players? What kind of environment will the application be deployed in? Does the product live up to the expectations set for it? Are there any legal requirements we need to be aware of? Each of these questions provides valuable information to help testers set their priorities on the right areas.

Fix the Show‑Stoppers, and Understand What Constitutes a Show‑Stopper

Many testers are trained to look at crashes as show stoppers, but they may not understand that a key business rule is being violated, or that a poorly worded paragraph or stray punctuation could prove embarrassing should it get out. Many testers would consider text issues to be a low level cosmetic bug. That may be true, but in this case, it may be an issue that really sets off an alarm in the customer buying the product.


The Perfect is the Enemy of the Good

There may be many small issues that have been found, and there may be many low traffic areas that may not have received thorough testing. What are the odds that area will be hit? What are the chances that the issues I have discovered will be seen as catastrophic? I may have to leave some small issues on the table, but I better make really sure that I have covered all the high risk areas After doing that, the project manager, development team and test team, along with any other key stakeholders, will have to make a judgment call. Different situations will inform those decisions, but there are times when “good enough” really is, and waiting for perfection will prove more costly than beneficial.

When we look to save money due do not taking additional time into the equation, it’s entirely possible that the money saved will be outweighed by the opportunity costs of the time we have lost. Take the time to determine what your “real wage” as a tester, as a manager, or as an organization, actually is. Consider what the value of your time actually is, and make sure to include that when trying to determine how to reduce the cost of testing. With an eye towards looking at both money and time, it’s possible to strike a balance where both can provide a positive return on investment.