BOOK CLUB: How to Reduce the Cost of Software Testing (20/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry. Ths entry covers Appendix C.

Appendix C: Rapid Test Augmentation by Jon Bach

Jon starts this chapter with the analogy that we outsource things in our lives all the time. When we call a plumber or go to a restaurant for dinner, we are actively “outsourcing”. Why do we do it? There’s usually two big reasons (though there may be several others). First, we need someone to do a job that we can’t easily do. Second, we get someone to do something that we could do, but they can do it much faster. So why do so many have a visceral reaction when we hear the word “outsourcing” in testing? It’s because we often equate the word with “offshoring”; using a different country’s labor (usually overseas from the United States) to do work we either won’t or can’t do without great expense. Man, the arguments I’ve had over the years regarding that topic could take up several posts, and really, the question is “is the argument even fair?”. We have no idea if offshoring our outsourcing is really lowering costs and raising value… or do we? Jon seems to think he does. shall we find out :)?

Prior to working for eBay, Jon worked for QuarDev, which meant he was a manager for hire. Why would you call Jon? Probably because you need something done quickly, with a solid expertise, and a budget to allow you to do that in a way your current team couldn’t at that immediate time (not that they were not capable, but they just couldn’t do what was needed in that sphere at that time under the circumstances that necessitated hiring Jon, right?). One of the things Jon is famous for doing is called Rapid Testing (his brother James Bach gets the credit for inventing the approach, along with active help from tester like Michael Bolton and Cem Kaner. Rapid testing is the “skill of testing any software, any time, under any conditions, such that your work stands up to scrutiny.” (James Bach, Satisfice)

Note, Jon’s team is not cheap. Some offshore serices can do “the job” for 5 times less. The real question though, and Jon is willing to stake his reputation on this, is that they can’t, not with the level of ability he an his team can do. Their philosophy is to make the sales conversation about value, not price.

As we’ve seen in this book, cost is entirely context based. One context doesn’t hold up to another. In this case, Jon is comparing labor costs and the fallacy that all testers are created equal, and there’s really little skill involved and one tester is synonymous with another (editors note; hang out with Jon for about an hour and test with him. I promise you will see the interchangeable testers fallacy is exactly that!).

An interesting development was taking place before the downturn in 2008 called “re-shoring”; projects that went overseas because time differences, language barriers, skills levels and other differences were significant enough that the cost alone was no longer the deciding factor. Many times, it made more sense for a value perspective to keep the work local. Sadly with the downturn, the flow seems to be going the opposite direction again.

To be a consultant means to go beyond the stereotypes and the bad jokes everyone has about consultants (jokes that tend to have a sting of truth to them). Jon’s strategy is to:

1. Charge a fair price.
2. Have some ideas about how to execute what you recommend.
3. Write reports that tell the story you need to tell.
4. Write simply.
5. Tell them what you found out and call them "findings."
6. Talk about what you hope to do next or how you see yourself being involved.
7. Discuss your code of ethics.

Questions to ask clients to help them get the best value out of the engagement:

1. "What's important to you?"
2. "What does success look like?"
3. "What's the worst that could happen?"
4. "What would you like to do less of?"
5. "What would you like to do more of?"
6. "What do you wish were different?"
7. "What problems are you trying to solve?"
8. "What's working?"
9. "What would you start doing, stop doing or keep doing?"

Jon likes to speak to the very top level of management, but also likes to spend time side-by-side with the testers to see how testing actually gets done. A lot is learned at both ends as ell as in between. Does the company value "heroism," and does that mean they they *rely* on heroes to solve major problems every time? Are there processes in place that prevent the heroes from becoming casualties? If so, are they actually applied? There’s a happy medium between heroism that’s excessive, and process that’s stifling.

Rather than polishing up a PowerPoint presentation, take out a sheet of paper during the pitch and ask: "Can we work right now on one of the problems you're having?" The way you probe for values - thoughtfulness, enthusiasm, and an earnest interest in the work – will help lead you to find the best fit for who you partner with.

Some Sample Questions FROM Clients 
Your bug database or mine?
Can I get the same tester as before?
How do you train?
Can I get resumes from your staff?
To what associations do you belong?
Can I see the templates you’ll use?
Can I customize the status reports you give?
If I need to postpone or cancel, what’s the penalty?
Can I talk to a tester in the lab?
What are your working hours?
Will you work overtime or weekends?
What’s your hiring process?
Will we be billed for the hours we don’t use?
How do you measure test coverage?
Why didn’t you catch that bug?

Some Sample Questions FOR Clients 
Your bug database or ours?
Can I talk directly to a developer?
What are your working hours?
Do you work overtime or weekends?
What’s your triage process?
When do you plan to ship?
Will we work onsite or here in our lab?
Do you use any existing tools that would be of help?
Can we see your existing bug database?
Did you want to devote time to regressions?
How often will you be giving us builds?
What are the minimum hardware requirements?
What kinds of users is this targeted for?
Has this been tested before?
Would you be a reference?

The “cost” of something isn’t just about price. It’s all of the context and values and weighting and considerations that go into the computation. Think of the average ink jet printer. They’re actually really inexpensive… the printer’s themselves, that is. The ink cartridges, though, can cost up to half the price of the printer!

List all of the things that matter to you – all of your hopes and ideals and values. Put them into a list and give a “gut-feeling” ranking to each of them on a scale of 1 to 10. Share it with the people on your team who are charged with making a decision. Add, delete, modify. But also stay alert to new context that emerges once the decision is made.

To know “the cost of testing”, you have to know the context of testing, the value of testing, and the questions of testing. You can buy a cheap car for $500 and maybe it’s just what you need. You can buy a new car for $50,000 and maybe it “pays for itself” in the benefits you get from it.

For the question of hiring a test lab to do rapid test augmentation, you may be able to reframe conversation to your stakeholders and decision-makers – helping them realize that you’re not hiring them to find bugs, you’re hiring them to assist you in gaining more visibility about the value you are offering to your customers.