Understanding Mental Modeling, Scientific Method and Design of Experiments: 99 Ways Workshop #16

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions.

Suggestion #16: Also get an understanding of mental modeling, the scientific method and design of experiments - Kinofrost

When we think of things in systems, and try to figure out how those systems interact, we create a mental model to help us interpret what we see. It's not complete, and it's not perfect, but it helps us with the ability to reason how things work, or how they might work. Often, taking the next step and diagramming out how systems work and the interdependency helps us expand or refine the mental model that we already have.  

The Scientific Method is part of the process of refining the mental model that I create, and it has some fundamental aspects that need to be followed to be effective. The Scientific Method requires the person using it to characterize a situation (I make a mental model of what I think should happen). This is called a hypothesis. As I develop that hypothesis, I consider the factors that might confirm it, as well as those that might refute it. Based on the determination of those factors, I then create an experiment, where the hypothesis is challenged. By performing the experiment and analyzing the data I receive, I can determine if the hypothesis is correct, if the hypothesis is false, or if more experimentation is required. The data and experiment are shared with others to review and consider, to verify that the experiment performed was valid and repeatable.

Workshop #16: Create a Software Test that Utilizes the Scientific Method

Since we are looking at this from a software testing perspective, I would suggest that the tester consider an issue and consider "how could I model my tests in a manner that is consistent with and uses the scientific method? 

Often, I use hunches or my own personal feelings when we determine that something is correct or incorrect. This points back to my previous post about understanding and using heuristics as oracles. There are a variety of situations where I have a strong confirmation. If I run a spreadsheet calculation where two and two returns a result of five, the history of math tells me that is wrong. There are also situations where I have a weaker confirmation. I just don't like the way something looks, or I don't think other people will like the way something looks. In situations that are aesthetic, the scientific method may not offer me much. Fortunately, there are plenty of situations where I'll have more concrete data to consider.

Search is interesting since there are a number of "rules" that can vary from application to application, as well as how it is implemented. Let's use Search as a model to set up an experiment that utilizes the steps of the scientific method.

Note: this may be harder to control with larger applications (Google, DuckDuckGo, etc.),  so I'll consider this within the domain of a dedicated application.

First, let's think of a variety of search terms (they can be words, phrases, wildcard searches, etc.). Let's make a hypothesis, which could be as simple as "I have 30 documents that have the word "JMeter" in them. If I enter the term "JMeter", I should be able to see those 30 documents".

Next, think of ways to prove the hypothesis is false. As a software tester, I prefer looking to see if I can disprove before I prove. If I'm too fixated on evaluating if something is true (that it "works"), I may subconsciously focus on just those areas where I confirm what I want to see. In this case, I would want to try to see if I could look for additional items in those 30 documents, perhaps using additional words or word fragments to see if I can locate specific JMeter documents, and exclude others.

I then run experiments in a variety of ways to see if the term I have entered will return something that does not correspond to the "desired result". If I get entries that do not conform to the search criteria, I write them down.

Finally, I share my data and consult with the developers to determine if the search code is wrong, if my search criteria is wrong, or if there's any "unseen elements that may be affecting my results. When we conduct experiments on a complex system such as a web application, there may be a variety of variables we have not considered, or may not even know about. Perhaps the search engine is configured in a way that we have not taken into account. As I learn of these additional variables, I then add them to my mental model of the system, reformulate my experiments, make new hypotheses, and try again.

This process is  repeated and refined (including changes to code and repeating tests) until I and all involved parties decide that the application in question is doing what it is "supposed to do". 

Bottom Line:

The scientific method, and structuring tests around it, is a way that we can be more specific and more exacting with what we are testing. In areas of look and feel, it may not be very helpful. In areas of data accuracy, calculations, returning expected results, applying security rules, etc. it is a very consistent methodology. Using it can help make sure that I am testing what I intend to test, and communicating my results in a way that is sound and will stand up to scrutiny.