BOOK CLUB: How to Reduce the Cost of Software Testing (17/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 3, which is sub-titled "How Do We Do It?". As you might guess, the book's topic mix makes a change yet again. We have defined the problem. We've discussed what we can do about it. Now let's get into the nuts and bolts of things we can do, here and now. This part covers Chapter 16.

Chapter 16: Rightsizing the Cost of Testing: Tips for Executives by Scott Barber

Scott makes the point early on that, unlike the other chapters in the “How Do We do It?” section, he’s not specifically writing to testers. He’s aiming for The CEO, CIO, or CFO and other senior execs that pull the trigger on decisions that they may really not understand. Even with that, the message here is still valid to front line testers. You may find yourself having a conversation with a senior exec, and you may be able to bend their ear and discuss this with them. Also, some day, you might be a high level executive, faced with the same issues. Wouldn’t it be cool to already know many of these details so you don’t make the same mistakes so many organizations do? Forewarned is forearmed.

For those who want to desperately believe otherwise, unless you sell testing services, there is no revenue in testing. There isn’t. It’s an operating expense like so many others, but it’s one that has an unfortunate detail going against it. Unlike some expenses that are quick to notice when they are changed, it may take several testing cycles to notice what happens if you cut testing from the budget. It may be months before the support calls increase, or before the cancellation notices start coming in, but in more circumstances than not, cutting testing is a case of “penny wise, pound foolish” and the result is usually a back tracking to real testing, and an even greater expense, until things normalize again. Testing has a disadvantage in the sense that, to most senior executives, unless they have spent time in the customer support area or in software development, testing is completely of their radar. We often joke that there are two experiences that come with being a tester. Either you are grudgingly appreciated because you find issues, or you are ignored when you don’t (even when the code is well developed and there’s little to really make headlines discovering. We are not celebrated when things are going good. We’re really only visible when things aren’t. Not to mention we’re often seen as the bearer of bad news, and really, who wants to hear that?

For me personally, I’ve been lucky in that many of the companies I have worked have been small enough where I had easy access to senior executives. I’ve also worked with companies that were significantly larger and where they indeed didn’t seem to have any idea what their testing organization was doing, and made decisions that seemed logical at the time but came back to bite them later on. If you are the executive reading this, perhaps a change in approach can help.

Executive Tip #1: Change the Question

Rather than asking “What value are we getting for what we are spending on testing?” instead ask “What value do we want to achieve through testing and how much are we willing to invest to see it actually happen??” Trying to calculate the ROI on testing is a lot like trying to figure out if paying for auto insurance is worth it. It may not be for years and years, but al it takes is one big wreck without insurance and the numbers change immediately! Instead of waiting for a “testing actuarial table” to appear, identify the areas you want to see your business increase value related to testing, prioritizing the areas and then determine an acceptable cost to meet those goals.

Executive Tip #2: Focus on Value to the Business

Most of the time, testing has been defined with the value to the project, rather than the value to the entire organization. How do we do that? We help the company pass regulatory audits, if they are relevant. We can provide defense against clams of negligence, faulty or bad advertising, or perceived breaks in service agreements. In short, good testing can help us not get sued (never a guarantee, but the likelihood of getting sued goes way down with vigorous testing). Testing can help the support staff deal with issues quickly and effectively. Testers can help provide stability between releases. Testing can help identify risk and areas where rick can be mitigated. Testers also work well as product trainers, as they tend to know the product better than just about anyone not on the customer support desk (and yes, that often means developers, too) Testers find discrepancies between requirements and what is actually being delivered. Testers detect issues earlier in the product lifecycle (especially if they are discovered before customers see them). Testers free up time for developers to focus on new feature development. Testers act as the advocate for the end user regarding usability issues and prioritizing issues from a customer’s perspective. The likelihood is that, seen in this light, there should be plenty of value identified in testing efforts.

Executive Tip #3: Distribute Testing Costs Carefully

I’ll confess I’m not an accountant, so much of this is outside of my area of expertise to comment on. Suffice it to say that, by spreading the costs of testing around among an organizations divisions and having the divisions be financially held accountable for the areas that they are responsible for, then testing costs can be placed in the areas that make the most sense for the organization and not impact other cost centers needlessly, and at the same time, each area can provide the support and financial backing necessary for testing to be effective across all divisions.

Executive Tip #4: Demand Accountability from Managers of Testing Programs

Executives have a need for information that they can quantify. Testing is often difficult to fit into that paradigm, but it can be done. Executives need to make clear what information they need, and why they need it. Testers need to make clear what the information they can provide actually means. Instead of asking for a particular measure or metric, executives that understand the value that testing can provide will focus on talking about, and trying to determine what value they are hoping to receive and by collaborating with testers and test managers, they can develop a metric or measurement that accurately represents that value. From there, the managers of the testing program (whoever they may be) need to know that they will be held accountable to that mutually determined metric, and that the testers know that they will need to modify their approach to ensure they can provide that metric.

Executive Tip #5: Keep Tactics at a Tactical Level

We’re all aware of the fact that we are often expected to do more with less, and how well we do more with that less determines if we get promoted or if we get sent packing. Scott points out that in his experience, more times than not, test programs are often cut by people who have little or no involvement or understanding of what the test team actually does. The only ones who can really determine what costs can be reduced without significantly degrading the value of the testing program is, well, the test team! How can they help this process? Well, the rest of the chapters in this book might certainly be a good place to start :).