Test Techniques and What They Actually Mean: 99 Ways Workshop #41

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

Suggestion #41: Learn how to use testing techniques such as boundary value analysis, equivalence partitioning, etc. - Amy Phillips

One of the things that is valuable in the testing world is to be able to articulate what you are doing, and share your information and data with others. to do that, it helps to understand the various terms and points of reference used by many in both the programming and testing disciplines. We'd like to think that we all speak the same language, but he truth is, we don't. 

Many of the terms that testers use are not necessarily understood by programmers. I had this experience a few years ago when I excitedly explained that I'd read up on this idea of "pairwise testing" and how it might help us shave off some repetitive tests. None of the programmers understood what I was talking about until I explained how it worked and how it was applied. At that point my VP of Engineering looked at me with a gleam in his eye and said "oh, so it's applying an orthogonal array"… to which I nodded, felt perplexed, looked up orthogonal array, and realized "yes, pairwise testing is a use of orthogonal arrays".

My point is, I was using a term, the programmers were using a different term, but we were able to communicate what we were looking to do once we both knew what each other were talking about.

Testing has several terms that, on the surface, seem daunting and out of our reach. Interestingly, for many testers, when they look deeper, discover that they have been doing the basics of many formal testing techniques without realizing it. They had the skills, they just didn't know what to call what they were doing. 

Workshop #41: Get in the habit of looking up any testing term that you are not familiar with, and compare it to your current actions. See if you have an "instinctive" knowledge of the idea (i.e. "you're already doing it"), then expand and apply more of the principles.

Yeah, I know, that one's a mouthful, but seriously, much of what we hear about test approaches are things that many of us already do instinctively.

Let's focus first on "boundary value analysis". The formal definition is rather imposing, but it really comes down to a simple concept. In a given capacity (a function, a field, an option to do something), there is a minimum allowed value and a maximum allowed value. Those minimums and maximums will vary, but we can start with that premise. If there is a minimum value, then anything that would be "less than" that minimum value would be outside the boundary. Likewise, anything larger that the maximum value would be outside that boundary. 

Think of a form on a web page. The fields in the form typically relate to values in a database, and are generally sized to what the database can store. 

Think of the example of a Last Name. It's likely that the field is made to put in a string of alphanumeric characters that would represent a Last Name, and require that one be entered (i.e. the value cannot be left blank). With that in mind, a "minimum value" will have to be at least one alphanumeric character. Anything less should trigger an error, probably from the browser, something like "hey, you left this field blank". In this case, we can't really enter anything less than zero characters, and if we look at the database table columns, we can see that the last name column maxes out at 50 characters. 

Between the minimum and maximum values, entering two characters or 49 characters, the system will likely treat them the same. That could be wrong, but generally speaking it's likely that those "in boundary" characters will all work the same way. By considering the values between 1 and 50 as being roughly the same, we are applying the idea of "equivalence".  Our "valid values" (one through 50) and "invalid values" (zero, greater than fifty, non-alphanumeric characters, etc.) are examples of "partitions". 

If we can determine where we transition from valid to invalid values, we can create small sets of data that focus on the transition points. Those values are going to be the most interesting from a behavioral standpoint. Characters inside the valid range or farther outside the valid range are likely to act the same way within their respective partitions (or at least, they should).

What if with 51 characters, something strange happens. What might that tell us? It might point out that 51 is a boundary condition. Anything beyond 51 will as well (most likely). So now we know that we have a boundary at "below 1" and "over 50". By doing this test, we can see that our web front end looks to allow the user to enter as many characters as they want to for the Last Name field. 

What can we do with this information? The simplest thing is to suggest that we use a MAXLENGTH option on the web form so that the user can only enter in a maximum 50 characters. This solves the problem (in this case) of people entering in more characters that the database will accept, and preventing a strange error that a user might not understand if an exception is thrown (or worse, the action is denied and the user has no idea why). 

The point is, testing at the boundaries and just outside of them are likely to yield the most information. Testing every conceivable value on one side or the other would, in most cases, not be of much benefit. 

Bottom Line:

So, with that explanation, are you put off by the terms "boundary value analysis" or "equivalence partitioning"? Show of hands, how many people can say "oh, OK, that's what I already do"? Using these terms helps us to create a shorthand for deeper concepts, and that's very helpful, provided everyone knows what the terms actually mean (and yes, this is a gross over-simplification, there's a lot more to this and many ways in which these two ideas alone can be applied).

If you hear a term used in testing (any term), and are not sure what it means, do some research, and see if what you find correlates to something you are already doing. If so, see if you can broaden your application. If not, take some time to learn about it, try it out, and see how it can help you better focus your testing, and how you talk about your testing. Then discuss these ideas with your fellow testers and programmers. Oh, one final thing... be alert to glassy eyes. If you see them, take some time to explain the concept you are referring to. They may get what you are doing, but not know what it's called, or they may use a different name.