Weekend Testing Americas #7: Smart Enough To Get Hired?

So this was a bit of a different approach to a Weekend Testing exercise. Albert Gareev and I had been commiserating over the fact that too often, after we had announced the testing mission and charter, testers would quickly take off. After a few minutes, someone would ask a clarification question, and it would make clear that we missed something and then, of course, those who stuck around to hear the clarification were able to work with that fresh knowledge. Those who went off to test, didn't have that fresh piece of news. Thus when we'd get together with the group for the debrief, invariably someone would comment on the fact that they didn't know that new piece of information was provided.

For this week's testing session, we wanted to try something different. Specifically, we phrased the challenge in a way that had little to do with the program itself, and more with the premise. We also said early on in the challenge that we didn't want to have people run off to test until they heard everything we were going to share. This had the desired effect of having everyone stick around until the mission had been explained.

The Game/Puzzle: Cross the River

Press the round blue button to begin. The goal is to get all eight people across the river. You must respect the following rules:

• Only two people can be on the raft at one time.
• Only the mother, father & officer can operate the raft.
• The mother cannot be left with the sons without the father (or she’ll beat them).
• The father cannot be left with the daughters without the mother (or he’ll beat them).
• The thief cannot be left with anyone without the officer (or there will be even more beat-downs).

Hey, don't look at me, I didn't make the game ;).


The Mission:
“I’m hiring staff for my IT department. I was told that this simple program will help me in finding the smartest candidates. Your mission: test the program and report how it suits my needs!”

With that, a number of the testers stuck around and really grilled us. What was the point of the mission, why did we think that a game or puzzle would actually help weed out the smartest testers? While the flaws of this system were discussed, Albert and I noticed that the goal we had set early on was working. The testers were not running off to test, they were sticking around and actively questioning the mission! 

We took the time to look at ways that we could question the claims of a product using a “focusing/defocusing” approach in testing. Along with this, we tried out some different approaches to discussing the problem. Albert would act as the "stakeholder" in the situation by iterating certain ideas and concepts by putting the text in quotes, such as "we have considered that, this is meant to help us automate the process of hiring as much as possible". This technique was used to make it so that it wasn't Albert or I leading the testers, it was the "stakeholder", and this way the testers would more readily question the claims. In the past, we had seen that if Albert or I had said that something was based on a particular approach, more times than not the testers would take us at our word. In this manner, they were much less likely to.

Another unique approach this time around was that we encouraged the use of hashtags in the chat session. the new one we introduced today was the #Danger hashtag. This was used to help identify potential hazards to the stakeholders during this testing session. If you'd like to see the Dangers discovered, as well as the #Issues we found with the process, please feel free to check out the chat transcript for details.