BOOK CLUB: How We Test Software At Microsoft (8/16)

This is the eighth installment of the TESTHEAD book club covering How We Test Software at Microsoft. This is final part of Section 2, which deals with the philosophy of testing and the testing concepts that Microsoft embraces and teaches its SDE’s and SDET’s. This section covers the area of creating models of working systems and coming to grips with making complex things more simple and making random things less random. Again, the density and detail provided could take several blog posts to cover. Rather than give a run down and bit by bit summary of the entire chapter, this is a high level view and will naturally only be able to skim the surface.


Chapter 8. Model-Based Testing 


Alan discusses the ideas behind model-based testing by making a comparison to Boeing and their method of designing a new jet aircraft. Since it is impractical to build scale versions of a jet airliner each time to test out an idea or approach to construction, much of the research and development is done using models, a method to help make potentially complex ideas more easy to understand or visualize. Mathematical equations are often challenging to children, but picture diagrams showing what happens help to simplify the concept and give the student another way to picture the solution and, later, have more confidence when they encounter the same kind of equation again.


We use models all the time in our work-day lives; when we draw a network map, when we draw the layout of a lab to see if we’ll have enough space to move a piece of equipment in, when we lay out the visible pieces of a GUI front end, and most certainly when we develop or test software code.


Model-based testing (MBT) is a method that Microsoft and other companies use to help outline their thought process or designs. Whether it be a flow chart on the back of a napkin or using full scale visualization tool like Microsoft Visio. The challenge is how to get test cases from a model, and this section is Alan’s approach to that very issue.


Modeling Basics


At the simplest level, a model is anything that describes a system. Behavioral models are said to have a starting state, one or more transitions, and an ending state.


A finite state machine (FSM) is the term used to describe a collection of states and associated transitions. FSMs are a natural way to express any functionality that is represented by states and transitions.

Testing with Models

Most testers use models and don’t even realize it. Every time we create a flow chart, or sketch a screen with notes, or block out the steps of a workflow, we are performing “unconscious” model based testing. Likewise, it is possible to create a model for many things, and getting a handle on the techniques can help us make better use of the approach.


Designing a Model

Alan uses the example of an 1962 AMC Rambler, and the beauty of the car was that at its heart it was a simple machine (I had a 1975 AMC Hornet, so I can attest to this age of automobiles being simple as a positive thing, but that may be better handled in another post :) ). The point Alan made was that the process of starting his car and getting it ready to drive, or stopping his car and turning it off, could easily be represented with a simple state diagram containing six boxes and a few lines to show which state the system was in at that given time (check the book to see the actual diagram). Of course models are limited views that allow us to see things in a simpler manner, by design they are not going to show every conceivable variation or option.

Modeling Software


A lot of software uses state-based testing, and being aware of what states are possible and what state appears at default can give you a lot of information about the application. In the book a simple application is presented, with buttons that allow the user to cause words to appear when certain buttons are pushed or clear the screens when another button is pushed. A very simple application, but helps demonstrate the point that the states of a system will help us determine what we’ve pressed or not pressed, and then ask questions of the application such as “how many times can I press this button?”


Building a Finite State Model

When a user creates a model, consider the following questions:

• Where am i? I need to know what state the application is in now, and I need to be able to describe (or know how to verify) the current state.
  
• What actions can i do? From my current state, what are the different things I can do?
  
• What happens when i do them? If I take an action, what state does it bring me to?

Graph Theory and Model Based Testing


Alan uses the example of the Seven Bridges of Königsberg, Königsberg being a city set on a river, connected across to the mainland and two islands by seven bridges. The question was asked if it was possible to walk a path that would cross each bridge one and only one time, yet still cross all bridges? Leonard Euler took on the challenge and proved it to be impossible, but in the process showed a way of mapping the problem and subsequently developing what we now refer to as “graph theory”.


The math used for traversing the bridges could effectively be applied to the states of software applications with each change being a crossing of one bridge to another location. Quite often, the tests that traverse states (like the bridges that cross rivers and connect islands), are likely places to find issues in code. Not every path is weighted the same, so being truly random may yield mixed results. Using a weighted traversal, however, allows us to take a mostly random path, but group tests in a way that certain paths will be more exercised based on their configuration.


Random Models

Alan refers to an example he calls “monkey testing”, where random or pseudo-random data is entered and the response is monitored to see if a bug occurs. The benefits of money tests is that they can uncover bugs with little effort in developing elaborate tests. The down side is that debugging the tests and determining exactly what course of action caused the error can be somewhat challenging. A combination of monkey and targeted tests make for a much more robust testing scenario.


Grammar Models


Grammar models are famously used with search parameters and regular expressions. These allow us to determine if a specific interface accepts the data it is expecting and generates an error message when it doesn’t receive the right kind of data (text in a date field, an email address improperly formatted, etc.)

A modeling success

Our test team used model-based testing on several new features from the Windows Vista operating system time frame and found that very extensive coverage could be achieved with a small amount of work. One of the hardest problems was creating a model that accurately depicts the system in question. Once this was achieved by properly defining all possible inputs, transitions, and outputs, we found it very easy to accurately define an expected result from any possible set of parameters. The model in essence becomes the ideal representation of the system you are testing and any deviation should be readily apparently.


We found model-based testing very robust because we could apply it to test areas ranging from APIs to UIs. This allowed us to do away with manual testing completely when the models were implemented. In particular with our UI, this saved us countless hours of manually verifying its functionality for all possible actions (button clicks, text inputs, and so forth). When properly scaled, our model-based tests could provide coverage ranging from quick verification tests to full-scale functional tests. This was done by paring down our list of inputs, transitions, and outputs to the subset that we were interested in. It would have been very difficult to achieve such a wide range of coverage if we relied on developing specific test cases one by one.


One of our successful implementations of modeling involved testing a new API that shipped out to developers in Windows Vista. Rather than write test applications that used the API or script a series of possible scenarios, we used modeling to generate a state machine for all possible function calls that a developer could make. This means for a given function call with given parameters, we could determine how all other subsequent function calls would react. We realized the true value of our framework in testing whether the API reacted correctly to different options/parameters/completion schemes without writing one-off applications for each interesting scenario. Our test footprint was much smaller with the model while achieving more coverage compared with our older approaches.

-- Jim Liu, SDET, Windows Networking


Model-Based Testing Tools at Microsoft 

There are a number of software tools that can help in the process of modeling the system under test, but again, a tool without proper understanding can be either ineffective (in the mildest sense) or downright dangerous (in the worst sense). Understanding how to create the models is foremost in importance, but even simple models over time and evaluation get added to, and grow in complexity. When that happens, just drawing it by hand may become more than the tester can handle, so Microsoft has made some tools to help the process. The first tool that Alan mentions is called Spec Explorer. This product was used extensively by the Hypervisor (Hyper-V) team. It’s since been packaged into Visual Studio and can be used by anyone.

Spec Explorer and Windows 7


In the upcoming release of the Windows operating system, known as Windows 7, the problem of providing a new managed service account class in the Active Directory directory service for automatic password management and simplified Service Principal Name (SPN) management was addressed by a new feature: service accounts.

[...]

There is a large selection of tools available for MBT. We chose Spec Explorer because of features such as C#-driven model development, integration with Visual Studio, and support for model slicing (using scenarios). A first model was created by using the example model shipped with Spec Explorer. Later the model was expanded by adding new states and transitions as well as additional parameter permutations. With Spec Explorer, you can add executable code as implementation of state transitions. Our model generated an XML file of test cases that could later be consumed by a traditional test execution engine.




Applying MBT for service accounts was very successful. Tests generated using MBT gave us greater coverage with less testing effort. Also, the models were easily expandable. It also helped us find interesting design issues very early in the development cycle.



One important lesson we learned is that, as a good modeling practice, we should start by modeling the most basic functionality and incrementally expand on it; this makes correction of modeling errors much easier. Following are some of the perspectives we gained by applying MBT versus traditional test design:

• MBT gives a different point of view on design of a feature.
  
• Tests are generated automatically with assurance of full coverage of the model in least steps.
  
• Extending the model is easy and takes advantage of previously generated tests.

To me, using MBT was fun and a great learning experience. It was easy to understand the model and gave me confidence about the extent of my test coverage. Interesting bugs were also found including design bugs, input validation bugs, and scenario bugs. I’ll definitely use MBT for my next feature.

—Sasha Hanganu, SDET, Windows Security

Modeling is important, but just as important is to also follow on and develop tets with and around that model. Models make it easier to understand and explain complicated systems, and therefore allow testers to better manage risk and hone in on areas where bugs are likely to be. 

Models can help users with random path discovery and make it possible to recreate and make the randomness easier to follow and duplicate. Models can help testers get a better understanding of where automation can be applied. Again, models are not a be all and end all, they are a tool, just like so many others that the tester has at their disposal. They are helpful in skilled hands, but they don’t do the work for you. As is key in every chapter so far, only a skilled, knowledgeable and aware (sapient) tester can do that.