. This is a continuation of Section 2, which deals with the philosophy of testing and the testing concepts that Microsoft embraces and teaches its SDE’s and SDET’s. This section covers the area of Structural testing, and again, the density and detail provided could take several blog posts to cover. Rather than give a run down and bit by bit summary of the entire chapter, this is a high level view and will naturally only be able to skim the surface.Chapter 6. Structural Testing Techniques
Bj starts out this chapter with an anecdote that deserves to be left as is:
A kindergarten teacher asked her students, "Who knows the color of apples?" A student dutifully raised his hand and the teacher said, "Ethan, what is the color of apples?" Young Ethan proudly stood up and stated, "Apples are red." Then, several other hands raised and the teacher picked another child. "Yes, Emma?" queried the teacher. Emma replied, "But some apples are green." "Correct!" exclaimed the teacher. Then, several more hands went up, and the teacher again selected an anxious student who was vigorously waving his hand. But young Caroline couldn’t control herself and before the selected student could answer she shouted out, "And some apples are yellow." "Yes, that’s right! Apples are red, green, and yellow," stated the teacher. But before she continued with the lesson, the teacher noticed a quiet little girl in the corner of the room with her hand still raised. The teacher called upon the child and the little girl said, "Apples are also white." The somewhat puzzled teacher politely replied, "Elizabeth, there are red apples, and green apples, and yellow apples, but I have never seen a white apple." The other children in the room began to laugh. Elizabeth stood up, peered over her glasses at the teacher and bluntly declared, "All apples are white on the inside!"
This is a great analogy, because very often we look at our applications from the external view, and our testing is focused primarily on what we see and how we interact with it. As the girl in the Kindergarten class accurately points out, the skin is less than 1% of the total volume and consistency of an apple. It’s the same with software. What happens underneath the skin is every bit as important as what we observe on the surface (if not lots more so).
This is where structural testing techniques come into play. Structural testing is necessary because the surface level behavioral tests and the direct function level unit tests don’t cover everything. There are lots of things that are not even touched by these two approaches. Structural testing is a white box testing approach. Intimate knowledge of the code is needed to make it effective. This means if a program is coded in C# or VB, it’s important that we also be very conversant in that language as well.
The big question Bj confronts, and one I’ve long wondered about, is “do testers who have intimate knowledge of the code end up biasing their testing to the functions themselves, meaning they prove that the functions do what they are meant to do, and that’s it?". I’ve long feared that I would be biased in this way, but Bj makes the case that here is where test rigor and focus is required, and the same exploratory mindset that is used for behavioral and black box testing methods should be applied here as well, and if they are applied, then those biases will likely not come into play.
Bj makes a comparison to testing done at Microsoft over several years, and has determined that when scripted testing and exploratory testing from a behavioral and black box perspective are applied, the difference in issues discovered by percentage and code coverage provided are relatively the same, with a nominal increase in coverage by combining both methods. Using structural testing in addition to behavioral methods brings the tester to close to 91% test coverage. Key takeaway: test tools are tools, and each does their jobs differently. To get the most out of them, make sure you are using the right tool for the right job at the right time. Structural tests will give you clues that no scripted or exploratory testing will hint at.
Block Testing
I have to admit that this is about the point that I feel myself getting out of my league, but I found the descriptions for talking about block coverage vs. statement coverage to be actually understandable (of course, that may just be repeat exposure, too). Statement coverage basically means the number of statements or lines of code that have been executed in a given round of testing. Block coverage means tests that measure groups of statements that don’t branch. Flow control mechanisms and the decisions made determine how much coverage is actually provided. In short, it’s possible to execute all statements, but not actually cover all of the blocks, and vice versa. You have to be aware of the differences to make sure that you are covering what you think you are covering. Even then, statement coverage or block coverage does not necessarily mean effective tests. This is where that nose for trouble that makes exploratory testing such an effective technique can come in handy while devising structural tests.
One place where structural testing really shows its benefits is with error control or exception handling. Black box/behavioral testing methods do not typically give much insight into where to find them, or to guarantee they have all be loaded and shown at least once (referencing “Whittaker Attack #1” again). Therefore, knowing what it takes to trigger these exceptions and generate errors is an important part of structural testing.
Decision Testing
Decision testing takes simple Boolean expressions (yes or no, true or false, 0 or 1) and branch one way or another based the condition being true or false. goal of decision testing is so that the tester can verify both true and false outcomes of Boolean expression. The benefit of this type of testing is that flow control can be closely monitored and branching can be observed based on a simple yes/no answer. For more complicated conditions, or multiple conditions, that’s where Condition testing comes in.
Condition Testing
Condition testing comes into play when there are multiple options to evaluate, such as AND or OR clauses in statements, where some things may be true while others are false, and containing two or more Boolean expressions. Condition tests evaluate true and false outcome of each subexpression within a compound conditional clause.
Basis Path Testing
Path testing tries to cover and exercise every possible path through a program. In reality, the sheer potential number of tests, especially in code paths that incorporate nested loops, can become astronomical. Performing exhaustive testing is perhaps not impossible, but it’s incredibly impractical in many cases, especially those that end up with trillions of test cases that would take a thousand years to complete, even at their top speeds.
Bj takes a diversion here to describe what cyclomatic complexity is.
Cyclomatic complexity measures the decision logic used to control flow through a module or a function. The formula to calculate cyclomatic complexity is v(G) = Edges – Nodes + 2, but if you must calculate cyclomatic complexity by hand an easier formula is to simply count the number of conditional clauses and add 1 (v(G) = p + 1).
A basis path is defined as a linearly independent path through a function. For example, in a function with one conditional clause there are two independent paths through it.
The simplified baseline path technique is described and applied in the following steps:
1. Identifies the shortest baseline path (the path with the fewest number of conditional clauses) through a function from the entry point to the exit point of the function being tested.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow the shortest path from that conditional clause to the exit point.
6. Repeat steps 2 through 6 until all basis paths (equal to the cyclomatic complexity metric) are defined.
While the simplified baseline path technique can identify identifying your basis paths, it can also result in a set of paths that are either too restrictive or otherwise considerably unlikely. To address this there is also the Practical Baseline Path Technique, which differs in the following ways:
1. Identify a likely functional baseline path through the function that is representative of highly probable control flow through the function that is most commonly encountered at run time, or the most important or critical path through the function.
2. Return to the entry point of the function.
3. Trace control flow from the entry point to the first conditional clause that has not been evaluated to both its true and false outcomes.
4. Change the conditional clause to its alternate outcome.
5. Follow a path to the exit point that includes the maximum number of conditional clauses traversed by the baseline path.
6. Repeat steps 2 through 6 until every conditional clause has been evaluated to both its true and false outcomes and all basis paths are defined.
Basis path testing requires each outcome for each conditional statement be tested independently. In the event there is a flow control that has an option for a yes or no answer, basis path tests will require the user to follow those commands and the structures they lead to, including tests that bypass code structures. Net result, more testing than block or decision testing will provide on their own.
Key to this whole discussion is, of course, the fact that this approach to white-box, code aware testing is a technique, and it’s one of many. It isn’t the be-all and end all technique, just as Exploratory or Scripted testing aren’t the be all and end all either. Context, timing and focus all come into play. Structural techniques are great when the focus is code, statement and block coverage to make sure that you are addressing areas that black box testing will not give you clues about. Risk goes way up if we don’t actually realize that we are traveling down pathways that we would not know to test or even understand would need to be tested without that level of knowledge. Make no mistake, any white-box testing technique takes time and patience to master (again, I freely admit this is an area I need help in, and I appreciate the detail in which it was covered).
Chapter 7 coming this Friday.
No comments:
Post a Comment