TESTHEAD REDUX: My "Software Testing" Toolkit

One of the things that I have tried to do over the past two and a half years is look at what I have learned and determine, ultimately, what is worth my time and attention. It should be no surprise that there are things that I would learn, or start doing, that would eclipse older habits or activities.

In this vein,  I decided to go back and review old posts. Could I update them? Did I still agree with them? Have I totally abandoned some of the things that I used to do? One of my first entries (more specifically, my third), was called "My Quality Assurance Toolkit", and was an honest attempt to explain the tools and the things that I considered important to my work at the time.


What has changed since then?

The first and most noticeable change is that I no longer carry a "Quality Assurance" toolkit. The past two and a half years have shown me enough times that there is no such thing as "Quality Assurance", at least not in or around my sphere of influence. Instead, today I carry a "Software Testing" toolkit.

Then: Test Plan Templates
Rationale: having a few ready-made templates to use for certain circumstances can cut down on the time needed to write a first pass test plan.

Now: HTSM Mindmap (and XMind), Test Heuristics Cheat Sheet (Mug), Rapid Reporter
Rationale: Seriously, I have not written a traditional test plan in over a year. The reason? I work in an Agile shop now and, at least in this current environment, I don't need to. Now, before anyone stops to think that I don't use guides to my testing, I do. I use different guides now, more based on exploration and less based on scripted test cases. I actually use three tools for this. The first one is a mind map that I made from James Bach's Heuristic Test Strategy Model. I use this whenever I want to go through and examine a new piece of functionality or get to know a system that I'm unfamiliar with. In addition to just traversing the map, I also write down observations in the map for future reference. The second one is a coffee mug that sits on my desk. It's my Test Heuristics Cheat Mug, given to me by Elisabeth Hendrickson, and encapsulates all of her Test Heuristics Cheat Sheet (actually, it's never had any beverage in it... it holds pens, and sits just beneath my monitor screen so I can spin it around and get fresh ideas any time of the day). Additionally, I also use Shmuel Gershon's "Rapid Reporter" tool for recording observations during test exploration. Alas, it's only available for PC at the moment, but when I test on my PCs, it's a beautiful thing to use to record and capture my test ideas and meanderings.

Then: A Test Dashboard
Rationale: A simple dashboard will give you a heads up in how to communicate what you are doing to others in your organization. What matters is that you are communicating the relevant information in them and that your data tells a story about what you are seeing.

Now: Cucumber
Rationale: A test dashboard makes a lot of sense if you are in an environment that counts test cases and cares a lot about pass/fail rates. Now, I look at stories and their acceptance criteria, and I write test cases in Cucumber that examine that acceptance criteria, and show the pass/fail rates that way. More to the point, I tell the story in a different way now. Instead of giving values such as "well, I ran 700 tests, and of those 700 tests, 690 passed and 10 failed", I now focus on explaining what I have learned from observing these test runs. Qualitatively. Sadly, dashboards are not really good at telling a qualitative story.

Then: Cygwin
Rationale: There's something to be said about the quick and direct elegance that UNIX pipelining of utilities provides, so cygwin is my simple compromise.

Now: OSX, Ubuntu, and Cygwin
Rationale: I currently work in a web services world, one that is built on Rails. The development machines are Macs running OSX. Our users could be running anything. Thus, it's important to be able to spin up any environment to test with. Right now, that means VirtualBox on a Mac, as well as a couple of standalone PC laptops, but the need to have Cygwin as a necessity on a PC is lessened. It's still a great set of tools, but if you have access to a dedicated UNIX platform as your primary machine (as well as Dropbox for storing results files), it's no longer such a necessity.

Then: A Scripting and a Development Language (C#, C# Script, Perl)
Rationale: At the time, C# and C# Script were the languages I was trying to learn to work with better, with Perl as my back-up scripting language (mostly from having done CGI for web servers years earlier).

Now: A Scripting and a Development Language (Ruby, JavaScript)
Rationale: As I have had the advantage of working in a shop where my tools are the developer's tools, I have taken on the challenge of trying to learn as much of their programming methodology as possible. Many tools are used in our environment, but Ruby and JavaScript are far and away the most used.

Just like before, getting up and running with scripting isn't hard, but getting proficient with any language (programming or scripting) takes considerable time and effort. I'm still working on the proficient part.

Then: An Active Library Card
Rationale: The ability to get access to books on just about any discipline fairly quickly is very valuable. However, the true gem in my library card is the subscription that it provides to Safari Books Online.

Now: Blogs, GitHub, and an Active Library Card
Rationale: Though having a library card and access to lots of books through the library system, I have found that, at this point, most of the books I have not yet read are considerably out of date, at least the print versions. The Safari Books Online listing is good, but it's limited to the last two years, and thus has a limited listing of titles. For established topics, this is fine, but for emerging or developing topics, blogs are much more helpful, as are going directly to the GitHub repositories for open source tools that are currently available and reading what has been written for and about them.  For a good start on the blog front, looks at my blog roll; there are several dozen more where those came from. Check out Testing References for a more comprehensive list.

Then: Some Open Source Test Tools
Rationale: Developing some familiarity with an open source tool or a number of open source tools will give you an edge and provide you the ability to test things "right now". These tools can range anywhere from basic macro recording tools for applications to sophisticated web based capture/playback tools rivaling features offered by the professional offerings.

Now: Lots More Open Source Test Tools
Rationale: If nothing else, this is the one area where little has changed except me doing a lot more of it. Selenium/WebDriver, Watir/WebDriver, RSpec, capybara, Cucumber, and a host of other supporting tools have all made their way into my workaday world, and I find them very helpful. I see my involvement with these tools progressively increasing over time.

Then: Some Basic Reference Books
Rationale: "Testing Computer Software", written by Cem Kaner, Jack Falk and Hung Quoc Nguyen. "Effective Methods of Software Testing" by William Perry. These have been my go to books for years. Additionally, I felt it valuable to have books that were helpful with developing Domain Knowledge for the area I was testing.

Now: Blogs, Twitter, BBST and Some Basic Reference Books
Rationale: Today, my book choices have changed a bit. The testing book that I consult most frequently now is "Lessons Learned in Software Testing", written by Cem Kaner, James Bach and Bret Pettichord. I also like "Everyday Scripting with Ruby" by Brian Marick and "Perfect Software: and Other Illusions About Testing" by Jerry Weinberg. There's also "How to Reduce the Cost of Software Testing", but that's me being a bit gratuitous, since I helped write and edit it :). The real big change is where I'm now finding my testing inspiration, and it's coming less from dedicated testing books and more from books about history, philosophy and general sciences. One of my go to books now is "The Day the Universe Changed" by James Burke. I've also enjoyed titles such as "The Structure of Scientific Revolutions", "Philosophy Before Socrates", "Who Killed Homer?", and "The Social Life of Information", among many others. The point? These are not "testing books" per se; they are thinking books meant to challenge the way that we view the world we live in, and how we interact with it. Again, much like the variety of books, I've found that dozens of blogs and a Twitter stream tightly focused on software development and software testing make for more of my current reading as compared to the older reference books. I'd also be remiss to not mention the huge library of resources that is included and referenced in the BBST series of courses. Two and a half years ago, I didn't even know they existed. Now, I help teach them... that's a change for the better (and extra points if anyone remembers the commercial from the 70's that that comes from :) ).


Then: A Virtualization Library
Rationale: Having the ability to build and update environments on the fly can be a huge benefit.

Now: A Virtualization Library
Rationale: The exact same. Having the ability to play "what if" and not worrying about destroying my application, machine or network is hugely valuable to me. The fact that I can actually carry these environments with me now on a thumb drive is even more mind blowing and cool!


In addition to this, tools like Skype, Typewith.me, and CorkBoard.me deserve special mention. Perhaps the most import part of my Software Testing Toolkit, though, is the one that's the most encompassing, and by the power of the Internet, available anywhere... this blog. It acts as the repository of my hopes, dreams, frustrations and triumphs. What's more, it's an encapsulation of my ideas and efforts, both good and bad, and shows, I believe, the body of knowledge that I possess and practice, hopefully with considerably more emphasis on the latter.


How about you? How has your toolkit changed over the past few years?

Photo: http://www.fine-tools.com/G309550a.htm