BOOK CLUB: How to Reduce the Cost of Software Testing (11/21)

For almost a year now, those who follow this blog have heard me talk about *THE BOOK*. When it will be ready, when it will be available, and who worked on it? This book is special, in that it is an anthology. Each essay could be read by itself, or it could be read in the context of the rest of the book. As a contributor, I think it's a great title and a timely one. The point is, I'm already excited about the book, and I'm excited about the premise and the way it all came together. But outside of all that... what does the book say?

Over the next few weeks, I hope I'll be able to answer that, and to do so I'm going back to the BOOK CLUB format I used last year for "How We Test Software at Microsoft". Note, I'm not going to do a full synopsis of each chapter in depth (hey, that's what the book is for ;) ), but I will give my thoughts as relates to each chapter and area. Each individual chapter will be given its own space and entry.

We are now into Section 2, which is sub-titled "What Should We Do?". As you might guess, the book's topic mix makes a change here. We're less talking about the real but sometimes hard to pin down notions of cost, value, economics, opportunity, time and cost factors. We have defined the problem. Now we are talking about what we can do about it. This part covers Chapter 10.

Chapter 10: Cost Reduction Through Reusable Test Assets by Karen Johns

How much time do we end up wasting because we have to do rework for tests because of constant changes; keeping a test bed viable when, with a new version of the code, login processing and item numbering have changed. So we need to rework our test cases and our test data. ARGHH!!!! Painful doesn't even cover it. Could this have been avoided? Yes, it's possible, but it requires a little coordination between the development and test team and the idea of reusable testing assets. If we created login details and item numbers as centrally maintained and reusable data items, changes to development could have been applied to the test data as well. Does that seem like some sort of weird pipe dream? Well, it's not really. In fact, this is exactly what Karen is suggesting test and development teams do.

If we look at testing, it's a prime candidate for the creation of reusable artifacts. Test plans establish scope, responsibilities, environments, tools, approaches, procedures, etc. Tests usually consist of actions that input data or perform some interaction with the system, and return an expected results against which the system is verified. Test data describes the specifics of any values being input or expected from the system. It's likely to confirm functionality, you would use values that create failures so that you an determine if the system handles invalid logins, posts errors under certain conditions, etc. The data to provide these actions can be easily stored centrally. Should requirements change, rather than regenerate all of the data, simply make sure the data is updated and reformatted to meet the new requirements. No duplication of effort or many hours of modification necessary.

In many systems, the data that we produce can be reused to test a variety of scenarios. In examples of structured work flows, the variety of data to confirm functionality may be surprisingly small. I have experienced the benefit of having test data be part of my own scripts used for testing. With Cucumber, the ability to create simple Scenario Outlines with Example test data makes for a quick and effective way to create reusable tests and also provide re-usable test data that can be expanded or removed fairly quickly. Granted, this approach doesn't work as well for very large data sets, but for examining functionality with a few parameters, it's quick and effective.

Keyword testing is a method that a number of testing tool vendors allow to help structure testing an scripts in a way to offer reuse of objects and data. I've used this technique with tools like TestComplete, where keywords can be associated with steps in the process and with pieces of data. With a bit of modification, these tests can be data driven, and that can allow for a significant amount of reuse. As I mentioned previously, Cucumber is designed to allow for statements to be tied to particular underlying rules and actions (I use RSpec and Ruby for this purpose, but Cucumber also works with a number of different languages and frameworks). The benefit is that, through utilizing various step files and configuration files, we can create test data that can be reused and the process of updating the test data integrated with the development assets.

Karen discusses the idea of Full Reuse™ where tests and test data are designed so that they can be managed as reusable assets. As is common in many environments test data may be read in from an Excel spreadsheet or a CSV file to drive tests. This is a start to test reusability, but there is a high maintenance costs for this data should it need to be refreshed, updated or converted to a different format. If instead, a database of test data were maintained, then the changes to underlying structure, as well as adding or modifying the data itself, could greatly reduce the time necessary to maintain multiple data sheets for various tests. One database to store it all, so to speak :).

Full Reuse™ lets testers define reusable actions for the tests that they need, store them in a test repository, and then have a framework call on that repository to structure their tests. As new releases are ready for testing, the repository can continue to grow, enabling more and more reuse. Data templates can be developed and modify for specific tests without the need of having to duplicate the data itself. It's an idea that anyone that has ever done data driven testing already understands and can appreciate. Tools already exist to allow for this level of integration. It will require some tooling to get the process cleanly implemented, but once it is, the idea of structured test data and the use of data templates to drive tests can yield a significant savings in time of maintenance of test data. For small shops with just a few (or maybe even just one) tester, this is indeed a compelling idea :).