Try to Find Problems and Victims: 99 Ways Workshop #95

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #95: Do not try to find errors or bugs - try to find problems and victims. Testing is more than checking. - Thomas Lattner

Wow, that's a provocative statement, and it's one I agree with 100%. One of the things that happens a lot is that we end up couching "issues" and "glitches" into safe words like that. We don't talk about "problems" or "bugs" or "catastrophic failures" because they might have the potential to hurt someone's feelings. We are encouraged to report dispassionately, to stick with facts and to couch words in ways that are not inflammatory.


For the long term survival of our jobs and sanity, that may well be appropriate. However, when we test, drop the polite shtick. As Steve Martin quoted many years ago, 'Comedy is Not Pretty". Neither is testing, if it's done right. We don't have to speak to others like murderous pirates, but we sure had better do so to the code we hope to test, otherwise we will miss something important.

Workshop #95: Exercise the product you are testing with an air of "what's the worst thing I could do here?" Think of a person that could be harmed, and how. Do your absolutely diabolical worst to see if you could expose the most sensitive aspects of their data and exploit it. Then write it up (in whatever dispassionate verbiage you choose) and lay it on the table. See who stands up and reacts.

Yeah, these are getting a little more tricky to write here at the end, I will admit it. I already feel like I've said this several times in other workshops, but I'll say it again here. We do our best persuading when we can make the bugs we find personal, when the programmers and product team can most directly empathize with the pain. Therefore, set yourself up so you can really bring the pain (or give it your all trying).

- Create a persona.

- Make it as detailed and as data rich as you can.

- Give this person a back story, and as much "dirt" as you would want to keep hidden.

- Then do everything you can to expose that dirt (or have one of your teammates try to do it).

Some of you are saying I'm taking "bug hunting" and I'm creating an inversion. In a way, yes, that's exactly what I am doing. I'm approaching the application from the aspect that I want to learn all I can about that person, and I want to do so with whatever restraints I can configure. The more restraints, the more aggressive I want to try to overcome them. 

Can I determine a password? 

Can I sent HTTP requests that will send me back raw data in clear text? 

Can I get to their credit card information? 

Can I order things on their behalf? 

One of the oft heard phrases is "well, no user would do that". They are right; no normal user we have ever envisioned that would be friendly to our product would aim to do such things. We're not testing for nice people. We are testing to help us thwart truly rotten, obnoxious and dangerous people. If we could put a human face and emotion to the issues we find, we will get much more attention than if we have some abstract, corner case feeling bug. Think about it, what's going to draw your attention more, someone saying: 

"in this obscure case, where I entered in the same password 700 times, I was able to throw an exception and leave the machine in a bad state" 

or 

"hey, check this out! Running this looped bash script and cURL, I was able to clobber the machine, get to the database prompt, and I now have the credit card information of all our customers. Yee Haw, who wants a Harley?!!" 

Bottom Line:

Sometimes it's a little to easy to get into the mindset of focusing on the features we are testing, but neglect the risks that might reside around those features (and yes, that's a point I made in another post a few days ago). For many, though, as Roland Orzabel* so aptly put it in "Goodnight Song", "Nothing ever changes unless there's some pain." It's up to us to help our customers see the potential pain, and  prove that it is possible. Not an abstract pain, but one that has a person's name, face and anguish behind it. The more you can make the hair stand on end for those discussing your findings, the better :).

* Tears For Fears, Elemental, 1993