Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full
synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.
We are now into Section 3, which is sub-titled "How Do We Do It?". As you might guess, the book's topic mix makes a change yet again. We have defined the problem. We've discussed what we can do about it. Now let's get into the nuts and bolts of things we can do, here and now. This part covers Chapter 14.
Chapter 14: Science-Based Test Case Design: Better Coverage, Fewer Tests by Gary Gack and Justin Hunter
This chapter takes the idea of conducting test case design based on an approach called "Design of Experiments" (DoE). Gary and Justin propose how using DoE provides better and more efficient coverage compared to a comparable manual testing approach. The idea behind DoE is using “controlled experiments” that focuses on pairs. The example they used to illustrate the point was how 18th Century British Navy attempted to combat scurvy, and the experiments they undertook to discover that Citrus fruits were the best remedy. In this case the pair examples were defined by treatment and subject.
DoE in software test case design provides three benefits
– it guarantees level of “coverage” for factors; it ensures all 2-way, 3-way, or n-way interactions are included in the set of test cases.
- it ensures coverage is achieved with a “minimum” number of tests
- the effort to provide required inputs and generate test cases is much less than compared to creating test cases manually.
Most software defects (over 90%) are caused by interactions of 3 or fewer parameter values. So we should guarantee at least 100% coverage of 2-way interactions. Applications with high reliability requirements, such as medical devices or avionics, should guarantee a minimum of 100% coverage of 3-way interactions! That's a big challenge. How can we do it? With DoE test design tools, of course :).
There are terms that are used interchangeably. When I first tarted to use the term pairwise testing, none of the developers knew what I was talking about. When I explained it, a few of them said "oh, you mean combinatorics or orthogonal arrays!" Um, yeah, I guess… turns out now I understand why there is so much confusion. Pairwise implies only 2-way combinations, whereas combinatorial suggests n-way combinations. An Orthogonal Array indicates that, for each pair of columns, all combinations happen an equal number of times.
Some have taken issue with calling DoE-based / combinatorial methods as a “best practice”. Justin and Gary agree with them, but they also say they consider DoE to be at least a “good practice”.
There are limitations to the approach:
1. Requires the input of thoughtful test designers.
2. Testers can and will forget to include significant test inputs.
3. You should not rely exclusively on DoE-based methods as a single approach for all of your test cases.
4. DoE-based methods are generally not appropriate for unit or white box testing.
5. DoE-based methods are generally not well suited to find “race” conditions in multi-processor configurations.
6. DoE-based methods won’t necessarily find sequential dependencies.
7. DoE-based methods (and other methods as well) won’t necessarily find unexpected side effects.
To Learn More about Pairwise, Combinatorics and Orthogonal arrays (check the book for the full list):
- Hexawise blog: http://hexawise.wordpress.com/
- All-pairs Testing: http://en.wikipedia.org/wiki/All-pairs_testing
- Planning Efficient Software Tests (Madhav Phadke): http://www.stsc.hill.af.mil/crosstalk/1997/10/planning.asp
No comments:
Post a Comment