Wednesday Book Review: Beautiful Testing

Through the years, I have come across a number of books that I have used and valued. These may be new books, or they may be older ones. Each Wednesday, I will review a book that I personally feel would be worthwhile to testers.

Beautiful Testing is part of O’Reilly’s series of varied topics, with numerous professionals talking about putting into practice the various theories and tools that exist and sharing their insights with others.

Many of us who test as a dedicated profession realize two things. First, what we do requires a lot of emotional brain labor, and it isn’t easy to do. Second, we realize that there are a lot of people who do not share that view. Oftentimes, we have to go to great lengths to explain what testing provides for an organization and that (zoinks!), it’s not just the testers responsibility to focus on quality. In fact it’s the entire organizations role to focus on quality (repeat after me: testers cannot assure what is not there. Developers can create quality product and we can assure that the quality initiatives that are in place are effective, or we can show where quality issues are. We cannot manufacture quality out of thin air. Either it is already there, or it is not).

Beautiful Testing takes a great approach here, in that each chapter is written as a standalone case study. Each chapter likewise has a different author(s). Some of the details are very familiar to every tester, and some situations are unique challenges that many of us may not have faced yet . The first part of the book deals with testing as a people issue, and focuses on tester attributes and abilities. The second section of the book deals with test processes and procedures, and real world examples of those procedures. Part three deals with testing tools and how to make the most of them in real world environments and with real testing challenges.

What’s more, each author agrees to donate their portion of royalties for the books to a charitable cause. In this case, the charitable cause is “Nothing but Nets” a program to distribute mosquito nets in Africa to help stem the tide of malaria infections.

Chapter 1 : Was It Good for You? (Linda Wilkinson)

This chapter leads off the book and gives a great introduction to the mindset of a tester, and the reason and rationale they use to help a company get the most out of their software development time. It makes a clear case that “not just anyone can test” (or at least not do so and do it well), and it helps identify the areas testers really care about.

Chapter 2 : Beautiful Testing Satisfies Stakeholders (Rex Black)

There are many stakeholders that have a say and a personal vested interest in our testing being done well and providing a lot of information to help make good decisions. Those stakeholders range from customers, vendors and users, but also include such entities as law enforcement, elected officials, company shareholders and all of the other key contributors to any project (PM’s, developers, software developers, and yes, even our fellow testers).


Chapter 3 : Building Open Source QA Communities (Martin Schröder & Clint Talbert)

Using the example of Open Source projects, getting a community involved in the efforts will help get people excited about applications and give those who are part of that community a desire and drive to see it succeed. My own experience with this has been with the Selenium Users Group here in San Francisco. While I find using the tool itself to be interesting, getting involved with and getting to know others that are also actively involved gives me extra energy and motivation to learn and practice more so I can likewise share with the broader community.


Chapter 4 : Collaboration Is the Cornerstone of Beautiful Performance Testing (Scott Barber)

Scott shares some of his insights into the development of his approach to performance testing, and the idea that performance testing challenges can be tackled via collaboration with other groups.

Chapter 5 : Just Peachy: Making Office Software More Reliable with Fuzz Testing (Kamran Khan)

Fuzzing is described as a technique where deliberately corrupt data is entered into your application to see how the system reacts to the inputs (for good or ill). Kamran uses Excel as an example application and demonstrates using tools that fuzz input and data values.

Chapter 6 : Bug Management and Test Case Effectiveness (Emily Chen & Brian Nitz)

Emily and Brian share bug management techniques and methods defining defects as relates to their involvement with Bugzilla and the OpenSolaris Desktop development team.


Chapter 7 : Beautiful XMPP Testing (Remko Tronçon)

Remko walks through examples and issues faced with testing the Extensible Messaging and Presence Protocol (XMPP) and describes his approach to creating Unit Tests for testing protocol interactions.

Chapter 8 : Beautiful Large-Scale Test Automation (Alan Page)

Alan walks the user through an example of test automation on a grand scale, and shows that many of the approaches and methods that are used for small scale automation projects work the same way for large automation, but the scale is totally different. This chapter helps a lot in showing neophyte testers that the steps from one world to another need not be so scary.

Chapter 9 : Beautiful Is Better Than Ugly (Neal Norwitz, Michelle Levesque & Jeffrey Yasskin)

Python has made its way from an interesting yet obscure language back in the 90’s to becoming one of the go-to languages of the web and testing today. Testing an entire development scripting language puts a whole new area and emphasis on testing and stability.

Chapter 10 : Testing a Random Number Generator (John D. Cook)

Here’s a great example of taking an application that can be tested in a number of ways, and the correctness or incorrectness can be difficult to pin down.

Chapter 11 : Change-Centric Testing (Murali Nandigama)

Murali demonstrates a call system and makes the case that, instead of testing everything over and over again, make a series of tests that will focus on the change. By using a change-centric testing approach, the number of tests run nightly can be reduced dramatically.

Chapter 12 : Software in Use (Karen N. Johnson)

Karen describes the feeling and the responsibility of testing equipment that works in a Hospital’s Intensive Care Unit, the very definition of Mission Critical. This one hit close to home, as it described a situation my Dad (a retired physician) faced a number of years ago with a program and a glitch that almost cost patient’s lives in an infant ICU. Karen describes the process, ups and downs, and resolutions related to, in her words, working on a product that really matters.

Chapter 13 : Software Development Is a Creative Process (Chris McMahon)

Chris makes the case (and a really compelling one) that developing and testing software is artistic work. Evaluating software quality is evaluating art, and that, when we recognize the artistic aspect of creating software, Beautiful Testing becomes a reality.

Chapter 14 : Test-Driven Development: Driving New Standards of Beauty (Jennitta Andrea)

Jeanette introduces the idea of the Diderot Effect and relays it to test driven development and the unintended consequences of upgrading just one area of a process and thinking that it’s all done. To embrace the beauty of TDD, all aspects of the role and purpose of testing and embracing TDD have to be applied. Requirements, system design, he very act of writing code, the pace of work and the level of engagement of the testers involved all face changes when TDD becomes part of the landscape.

Chapter 15 : Beautiful Testing As the Cornerstone of Business Success (Lisa Crispin)

Anyone familiar with Agile Testing will notice the Mind-map that leads off everything, and gives a clear picture of the ideas that Lisa wishes to impart. The take away is clear, testing is part of the overall process of development, and testing is a process at every stage of development. Testing drives development, and development is not complete until tested.

Chapter 16 : Peeling the Glass Onion at Socialtext (Matthew Heusser)

Matt makes the point that, in mathematics, often the simplest solution is the most beautiful solution, and the same holds true for testing. Through examples at Matt’s company, Socialtext, he shows how they do not just test to show that they have done testing, but that the solution they have developed fits what their customers want to see.

Chapter 17 : Beautiful Testing Is Efficient Testing (Adam Goucher)

Efficiency and focusing on how to get the best bang for your buck requires setting some parameters, using some tools to help focus on the goal, and making a mindmap to capture test ideas and methods. Adam uses the mnemonic SLIME to help organize his approach ((Security, Languages, RequIrements, Measurement, Existing).

Chapter 18 : Seeding Bugs to Find Bugs: Beautiful Mutation Testing (Andreas Zeller & David Schuler)

Andreas and David discuss the idea of mutation tests, and the tool Javalanche to perform those tests.


Chapter 19 : Reference Testing As Beautiful Testing (Clint Talbert)

An inside look at how Mozilla tests the variety of products in the Mozilla portfolio, and how they create tests and their reference points. Their goal is to encourage people to get involved and test in the way that is the most simple, direct and easy to understand way possible.

Chapter 20 : Clam Anti-Virus: Testing Open Source with Open Tools (Tomasz Kojm)

A look under the hood at an open source product (Clam Anti-Virus, a tool I actively use and wholeheartedly endorse, by the way) , and all of the open source tools used to test it, along with the testing strategies used.

Chapter 21 : Web Application Testing with Windmill (Adam Christian)

Adam provides a quick tutorial in how to set up and use the Windmill web testing tool and a quick way to implement automated testing for web applications.

Chapter 22 : Testing One Million Web Pages (Tim Riley)

Tim describes the Spider and Sisyphus projects at Mozilla and how they use the framework to test huge numbers of pages and web sites.

Chapter 23 : Testing Network Services in Multimachine Scenarios (Isaac Clerencia)

Isaac describes the ANSTE test tool and how it is used at his company, eBox, to test environments with multiple and varying machines.

Bottom Line:

Not every section will be relevant to every tester, and I found a few of the sections not immediately applicable, but each section gives the reader a greater appreciation of the testing process in their respective sections. The multi-writer style for each chapter makes the book very engaging, and allows the reader to skip to the sections that matter the most to them. There is something for everyone in the testing process here, from technical testers with deep programming knowledge to relatively new testers without specific development backgrounds. Agile and traditional development methodologies will find value in these chapters, and overall it’s a fun read. If you are looking to put a little more elegance and art into your testing life, Beautiful Testing has a lot to offer.