TWIST # 13: Code Coverage with Zach Spencer

I’m still playing with ways to even out the sound in the interviews without causing Matt to sound like a John Lennon dub or squash the interviewee into sounding like they are coming out of a can. It just seems to be the nature of the beast when it comes to recording interviews via Skype and a single input interface. One of the changes I’ve made is to modify the Nyquist chain for Clean Speech to not render the audio in mono, since it will output it in stereo in the final version. Next week, we’ll see how it fares.



Today's interview is with Zach Spencer, a developer who made a conscientious decisioin to become a tester. He says he receives many comments of “oh, I’m sorry, what happened?” and respondes with what is for me easily the best line in the entire interview “I get WAY more interesting problems to solve than YOU do!” :). Zach talks about his work with Pillar Technology and how his understanding of development has helped him in testing, specifically glass box and gray box testing. Cyclomatic complexity and code coverage..

For those who want to check it out, here is Episode #13.

Standard disclaimer:

Each podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition, pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.

Again, my thanks to STP for hosting the podcasts and storing the archives. We hope you enjoy listening to them as much as we are enjoying making them :).

Wednesday Book Review: Rocket Surgery Made Easy

This book actually came from a recommendation from our upcoming TWiST interview. In it our interviewee was describing usability testing and suggested the book Rocket Surgery Made Easy. This book, written by Steve Krug, is a sequel of sorts to his book Don't Make Me Think, which is dedicated to the topic of Web Usability testing and optimizing sites to better to user experience.



While Don’t Make Me Think is the “what to do” book, Rocket Surgery Made Easy is the “how to do it” book. The key here is that this is not a long drawn out usability study example. This is doing quick and dirty usability testing in a fairly frequent manner. Several maxims appear in the book and are built into the prose and presentation. The steps he suggests are straightforward and quick to understand (the entire book is about 160 pages, easy to read in a day or less, and easy to implement tie ideas suggested in each chapter. Krug has a breezy and fun style to his writing, and plenty of self-deprecating humor to keep it entertaining, yet still focused on the main topic.



Rocket Surgery Made Easy is not meant to be comprehensive. Instead, it focuses on the ability to look at a design for a site, whether that is on a live set of pages or a design on the pack of an envelope or a napkin. The book guides testers to focus on the most important problems first. It also encourages developers to fix issues based on doing the “least you can do” approach by doing the least amount of modification possible (tweak rather than rewrite). By setting these tests to be a regular occurrence (“A morning a month, that’s all we ask” is the book’s first maxim), test teams can make usability and user experience a target for testing early and often, and keep looking at the product as its developing and ask “does it do what we really want it to do”? More importantly, it allows the customer to ask ”does this do what I really want it to do?”



At the end of the book, there is a sample script and forms that can be used for hosting a usability study and gathering feedback. Again, this is more of a workbook than it is a reference, though you can certainly learn a lot about usability testing from reading it. It’s not meant to be the definitive guide to all things Usability and UX. Rather it’s a method to get your feet wet doing actual usability testing and developing your own usability protocols.

Army of One: The Danger of Being a Silo

This weekend, some of my Order of the Arrow youth will be going up to an event called Conclave. That’s when the Lodges in our Region (8 total, covering a large portion of the San Francisco Bay Area and on east into the Central valley and down into Fresno) get together to participate in games, learning, food, fun, and service to their communities and to Scouting. It is in the role of Dance Team Advisor that I usually participate, but I will be at Wood badge helping train other adult leaders, as that’s the commitment I made close to a year ago to do.


It’s unfortunate that the events are being held at the same time, but there’s an even more unfortunate aspect… the dance team from Ohlone has elected not to participate without me being there. When I asked the team why they didn’t want to participate, they said it was because I wouldn’t be there. They didn’t feel like they knew the songs, the dance moves, or how to put on or care for the outfits if I was not there. Also, I made a promise to them after their solid performance last year that, if they wanted to do other things than dance this year, then they could, as they always had to give up participating in other events so that they could dance in the Pow Wow.


I realized with this revelation from them that I had failed them. I had assumed that what I was teaching them was being internalized and understood, not just what to do when I was there, but what to do when I was not there. This drove the point home that my knowledge was more of a “Silo” than I ever realized. Those of you not familiar with the term, a silo is a structure for storing materials like grain, coal, sawdust, dry cement, and other items that are needed in bulk. The key is that the items are kept separate from other materials until needed (in the example of a cement silo, the contents are kept away from water until it’s time to mix the cement with water and use it. It’s also possible top make mental silo’s, too. We think that our knowledge is something that we share with others, that we give away freely and think others are internalizing and will know what to do once they have it. This recent exchange taught me otherwise. My Scouts relied on me as a knowledge silo, and I helped them do what they needed to do on many occasions, but when they needed to be ready to go ahead without me, they were not prepared to do so.


In my testing life, I’ve come to realize that I take for granted the information silos that I have, and I lament when they are gone or unavailable. However, I need to realize that I am also an information silo at work, especially since I am an Army of One. Who can I share my knowledge with? Who is there to listen and pass on what I have learned? Would testing stop if I was gone, or would it continue on unhindered by my absence? While I certainly hope I’d be missed if I was gone (I wouldn’t need to be employed if they didn’t miss me when I was gone, right?), but it still means that there is some key information that I need to make sure that my team mates and project members know about. Tools like internal wikis, regular conversations with developers and the customer service team, notes about test tools and tests that have been written and automated need to be shared. Likewise, I share this blog with my team members, and while it doesn’t often share specific details about work, it does share some insight into how I think about these things (or at least I certainly hope it does).


As an Army of One, I’m expected to be self-contained and able to move and be effective whenever and wherever I can. That also means I have to be able to let people know where my arsenal cache is when I must be away. Sharing knowledge will not make us less valuable to our team. In fact it will ultimately make us more valuable because when we share, we solidify what we know for ourselves, and we then enable others to do the same thing. When the knowledge is too locked up inside of us, it’s ability to help others is greatly diminished, possibly to the point of being unusable to others because we haven’t prepared the way for them to use it properly. My Dance Team will get some better instruction from me over the course of this next year so that they can be more independent and not rely on me as directly. It’s my hope also in my role as an Army of One that I can also share my testing knowledge and skills with others so as to open up the possibilities for testing when I am not here.

TWIST # 12: Interview With Mike Dwyer

I learned a couple of new tricks with Audacity this time around that are helping me bring the time down to produce the show and turn episodes around. I realized that the envelope tool would allow me to make gradual changes to audio volume, but even more important, it could act as a virtual "flying fader"; no more separating tracks and leveling them independently required. I also discovered the chain command "Clean Speech"; it goes through and does a bunch of Nyquist batch commands to clean up vocal tracks. Most important, it automatically removes "dead air". It helps, but manual editing and paying attention to flow still has to happen, so I can't be replaced entirely by a software program... at least not yet :).


Today's interview is with Mike Dwyer, who Matt reverentially referred to as "a grown up in our industry" and one who admonishes us to not forget our past. We have such high turnover in software testing (and development) that each generation, it seems, needs to re-learn the lessons of the generation that preceded it. Mike has experiences with companies like Parker Brothers, Unisys, Wang and others, providing a great retrospective of quality and what it once meant in the heady days before the PC was an everyday item. Today, Mike works to help companies develop 'agile' methods and apply them.


For those who want to check it out, here is Episode #12.

Standard disclaimer:

Each podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to the podcasts and the entire library whenever you want to :). In addition,  pro membership allows you to access and download to the entire archive of Software Test and Quality Assurance Magazine (as well as issues under its former name, Software Test and Performance).

Again, my thanks to STP for hosting the podcasts and storing the archive. We hope you enjoy listening to them as much as we enjoy making them :).

What Are My Three Biggest Takeaways From AST?

Another AST class has come to an end. I had the privilege and pleasure to act as an assistant instructor for my second time during the BBST-100W session (that's shorthand for Black Box Software Testing Foundations class, the 23^rd session of that particular class being taught). This is my second time through the class as an assistant instructor, and the Foundations class is getting a reboot next month. Foundations 2.0 will begin in October, and I have signed on for another tour of duty as a "returning student" for its November run so that I can experience the new class structure, format and learning objectives. It should be an interesting experience, and I wonder how my experiences with the last three times through the class (first time as a Participant, second and third time as a "Staffer") will influence what I find of most value in the class.



I mention this because I had a chat session with one of the students from the most recent (and final) Black Box Software Testing Foundations 1.0 course. I'll not divulge who, as it's not important, but it brought up some interesting thoughts, and, well, that's the point of this blog… what do we hope to learn when we take a class or a seminar/webinar. Do we know what our specific objectives are going in? Is there a specific takeaway that we should take with us? This inquiry was started because I was asked what I felt the three most important things to take away and focus on from the class was (in addition to finding out about what their grade was and how it was determined). I answered that, honestly, I couldn't tell them what the three most important things were, not because I was being a smart-aleck, or trying to make things difficult, but because what I would consider to be the three most important things may not prove to be the three most important things for them.



I know what I took away the first time I took the class as a participant, and that was the importance of understanding how to give everything a critical read. In fact, this tripped me up so much, and proved so frustrating, that it was the ONE thing I left the class with above all other considerations. The questions are worded in such a way that, unless you have put in the time and considered the material, you will not guess your way to a right answer (Well, you could, but it would be very hard to do and you'd be very lucky to do it regularly). This is because being able to read critically, and to tease out the important aspects of the question, and then be able to actually answer the question, is a critical skill, and many of us are either lacking that skill or are not as developed in it as we could be. So that is still my number one takeaway.



The second takeaway actually came to me during my first time as an assistant instructor, when I helped staff the class that followed. As I saw many people fall into the traps I fell into, I wanted so much to just give them the answer, but I was advised to not do that. In fact, I was encouraged to answer questions with additional questions (you know, that Socratic dialogue that testers are all so famous for and that tends to annoy our developer friends to distraction (LOL!)?). The purpose was to not punish the students, but it was to make sure that they had a chance to think through the problem, and our questions could give them some hints as to how to find the answer without spoon-feeding the information.



The third thing that I found the most helpful and most important was the value of peer-review, and really committing to giving good reviews and being open to and willing to receive review feedback. This time around, I made it a point to answer and provide feedback to questions when feedback was lacking, but tried to keep a step back from providing too much feedback too early. The reason was that, in the first class I staffed, my review was often seen as though it was final (the "teacher" has made his pronouncement, and therefore all that follows is somehow less relevant). Let me make clear that I disagree with this, but that tends to be the outcome anyway. The quality of peer review went up when I decided to refrain from adding feedback until after everyone else had their chance to have their say, and most importantly, when the individual who originally posed the question had a chance to review and consider his answer based on the feedback as well. My keeping mum until they finished improved the value of the review and the review answers provided. IT also had the net effect that, for some of the questions, I really didn't need to provide feedback at all; the students did well enough and provided enough solid talking points that my comments would have been redundant.



So to my friend who chatted with me and felt like I didn't give them a straight answer, here it is. Out of BBST Foundations 1.0, these are the three most important things I took away from the class, and felt it most important to focus on. Notice that none of these is a specific testing technique. Testing techniques are nice side benefits of the class, but honestly, I don't think they are the most important things. Test techniques will vary from organization to organization, and much of the time, the techniques you use will depend on what your organization values. Those who follow a pre-determined script may find BBST Foundations uncomfortable; its very nature is that you throw away the script. What's good in one context may not be good in another; the three most valuable and key take homes in one person's mind and practice will vary from another person's key take homes. Thus for me, I think that practicing critical reading skills, asking probing questions to better understand the context of what you are testing and what's important, and the ability to give and receive solid and productive feedback are the three most valuable takeaways from BBST Foundations 1.0.



Ask me sometime in December what I think the three key takeaways will be for Foundations 2.0… I'm willing to bet my list may be different by then J.

Wednesday Book Review: Philosophy Before Socrates

I have to credit my company's VP of Technology for turning me on to this book. When I told him that I planned to go back and do a little "classical re-education" to fill some gaps regarding my "inductive reasoning" toolkit, he suggested this book as a good place to start.


"Philosophy Before Socrates" is exactly what the title suggests. It's a look at all (or at least a broad range) of the philosophers that preceded Socrates. Socrates, of course, is probably the most famous and most applicable of the Greek Philosophers to the testing profession because, really, testing is a continuous form of Socratic Debate. We pose questions and, through the answers we receive, we can eliminate those hypotheses that don't work for our purpose or help us reach our goal. Socrates, however, did not come fully formed and just appear. Nor did he invent Greek Philosophy. He tends to stand as the pillar of Greek Philosophy because we have a great deal of information about him, mostly through Plato's writings. By comparison, many of the earlier philosophers in the sphere of Classical Greece have little to no surviving records of their writings. There are a few fragments here and there, as well as numerous "testimonia" in others writings. In other words, we typically can only look at what others have said about these early philosophers to understand what they said and how they said it (if they actually said it, that is!).



It is into this strange dust storm of history Richard D. McKirahan bravely travels. It's also important to realize that the region we know of as Greece today only encompasses a small area of where the Classical Greek influence spread and was nurtured. The ancient area of Anatolia (which is in modern day Turkey) also figures into this story, as does the island of Sicily and the southern area of Italy. This book is dense, and to go through each and every chapter will make this review extremely long, so I will dispense with my usual chapter by chapter summary and group the ideas into larger areas.



The Sources of Early Greek Philosophy
The biggest challenge with the pre-Socratics is that we don't have much to go on that's original material. Remember, in the ancient times, if an author wanted to create a book, they wrote it themselves in long hand, or they had a scribe write it. When the book was finished, to make another copy required exactly that, another scribe to write in long hand an exact copy of the original, and so on and so on. The problem with this method is that errors creep in (usually unintentionally) and then we are left wondering if what Thales of Miletus actually said was what we have available to us today. In most cases, no originals exists of the early philosophers, and in many cases, only reference to these philosophers by other philosophers allow us to even know what those later philosophers say that the earlier ones said. In short, the biggest challenges is teasing out of the commentaries, fragments and "testimonias" provided by Aristotle, Plato, Plutarch, Cicero, Hippolytus and others the true words of the ancients.


Hesiod and the Beginnings of Greek Philosophy and Science
For many fans of Greek Mythology, much of the structure of the Pantheon of Greek Gods and Goddesses are described in Hesiod's Theogony ("Birth of the Gods"). While most of this work is orthodox Classic Greek theology, his other most famous work, the Works and Days, takes us into the workings of the agrarian yeomen farmer, and the world view that is in many ways seen as "ever so Greek", that of stoic effort, strength in the face of capricious gods, and the idea that one's life and success are measured by the extent and quality of fields and livestock, the harvest, and the qualities in oneself and one's family.



Miletus in the Sixth Century
What was it about this area that produced so many people that would challenge the conventional wisdom of their day, and set the Classical Greek world onto a new path of inquiry and reason? Part of it comes from where Miletus was situated. In many ways, Miletus was the Ancient world's "crossroads and cross naval paths" of the Western world. It was a sea-port, and as such it had access to trade from all over the known world, ranging from Greece to the Levant, Egypt, Punic Northern Africa and elsewhere in the Mediterranean. Miletus was what was referred to as a secular city state (or Polis). In short, Miletus was happy to keep its religious life and its political life separate. This was an attribute that would come to be seen as defining of most of the city states of the Classical Greek era and those in its sphere. Why Miletus was the first to flower with this "rational thought" is not certain, but many point to the fact that, because of its strong position in inter-continental trade, its lack of a dominance from a religious hierarchy, and its ability to have enough riches so that some people didn't have to scrape out a living from the soil alone, helped set the stage for people to look at the world and say "so, what have we got here?" in ways no other kingdoms or civilizations had the time or luxury to consider.



Thales, Anaxamander and Anaximenes of Miletus
Many people look to Thales as the start of the great Greek tradition of Philosophy. The fact that he was brilliant and unorthodox comes to us from various sources (Aristophanes mentions him by name in one of his plays written two hundred years after his passing). He comes down to us as being a bit of a "absent minded professor" type, but one who could also use the knowledge he gained to take advantage of the opportunities in the world, opportunities he profited from handsomely. In short, Thales enjoyed a position as a wealthy man, and that ability also allowed him time to make observations and put them into practice in ways a common man of the time would neither have the inclination or the free time to do (and have the influence to convince others that he was on to something). Thales would become famous for his statements regarding Astronomy, and the ability to fairly accurately predict eclipses. Thales is also believes to be the man that adapted Egyptian Geometry with numerous theorems, many of which Euclid was later to refine and codify.


Anaximander is credited with being one of the first known "map makers", having drawn the inhabited world that was, or at least the world that was known to him and to those who had traveled by land or sea in the regions they were aware of. Anaximander is credited with warning the Spartan king of an impending earthquake, and encouraging the kingdom of Sparta to seek shelter. He is also one of the first to look at space and see it as an infinite void where numerous stars are to be found, surrounded by an eternal substance that he referred to as Apeiron, which was distinct from all of the classic elements. It was from Apeiron that the world formed, and that various elements separated into their own classes from the Aperion in space. This is one of the first references to the idea that matter in space was formed and coalesced to develop the world on which we stand.



Anaximenes took the idea that all things come from Air, and that in his estimation, air condensed into fire, then into wind, then into clouds, then water, and finally stone, out of which all other elements can be extracted. One of the more dramatic and, at the time, unprovable aspects of Anaximenes theories was the idea that the Earth floated on air and that the rarified air would be strong enough to support the weight of the world within it. In short, Anaximenes was one of the forerunners who described atmosphere and its role in regulating and sustaining life.



Xenophanes of Colophon
Xenophanes was one of the first to break wholly with the idea that a universe was ordered and ruled by gods and goddesses. His poetry went after and attacked the statements made by Homer and Hesiod, and argued that the pantheon of gods described were irrational and disorderly, and did not fit within a defined KOSMOS as was developing in Western thought. In Xenophanes view, since all gods are defined by their region and by the peoples that believe in them, and since all gods cannot be different to different people and be the same, then this brings the entire pantheon of gods into question. Mind you, while there was a secular tradition in the Greek city states, this was a bold world view, and one that caused Xenophanes quite a few accusations of impiety. Xenophanes does not deny the existence of the divine, but he has qualms with the way that God was represented at the time. In many ways, the idea of an all present, omniscient deity that can be everywhere at once finds its philosophical moorings in Xenophanes. Likewise, for those who are interested in the development of epistemology, or the science of knowing what we know and how we know it, Xenophanes arguably deserves to be acknowledged, if not as the founder, then at least as one of its earliest known adherents. He makes the clear delineation between truth, belief, opinion and knowledge, and shows that they are not the same thing.



The Early Ionian Achievement
As the ideas of the four previous mentioned philosophers take hold, are debated, are considered, and in many cases discredited or accepted, we see that a population takes to the daring step to stop looking to tradition for all answers. In fact, the philosophers of Miletus and the surrounding area have specifically rejected tradition as an authoritative source of knowledge. The net result of this action is that theories must stand or fall on their own merits. This heralds the development of "rational criticism", and this rational criticism took place in the public square and was roundly debated by many individuals. Theories that did not hold up to observation were discredited, and new theories came to take their place. These in turn would be held up to scrutiny and likewise rise or fall. That's not to say that the early theories were all excellent and scientific. Many of them were what we would consider outrageous today, but the temperament of the time and place allowed for outrageous speculation, and in fact, welcomed it. The more outrageous the theory, the more interesting the debate would be. The idea of experimental science as was developed in the early Renaissance was mostly foreign to this age of philosophy; most theories stood or fell based on the rhetorical skills of those posing the ideas and the questions.



Pythagoras and the Pythagoreans
...and you thought that theorem about the hypotenuse of a right triangle was all he had going for him? Pythagoreas was the start of an entire movement related to not just the development of theorem proofs in geometry, but a whole range of opinions and beliefs including the transmigration of souls (mirroring in many ways Hindu and Buddhist reincarnation beliefs). Pythagoreus also wrote of an idea where the Earth was counted among the stars and circled a fire at the center of the universe, causing night and day. While the purpose of the "eternal fire" was said to be the house of Zeus, it does show that Pythagoras was one of the first to consider that the Earth may not be the center of the KOSMOS.



Heraclitus of Ephesus
Heraclitus is the first of the pre-Socratics of which substantial fragments exist, so comparison to his actual words and what others said about them can be made. The key idea that Heraclitus worked towards and expounded on was that of LOGOS, the idea being that speech and word, if properly attributed and formed, describe the nature of something (bedeviling as that thought might be in practice). In short, the only thing preventing us from truly understanding something is the language to describe it. If something is straightforward and simple, it is very likely wrong. Heraclitus summarized this by saying "out of all things there comes a unity and out of a unity all things" (and you thought Homer could be confusing!). Actually, for us testers who look to "context" for situations, Heraclitus is in many ways the forerunner of context-driven inquiry.



Parmenides and Zeno of Elea
This small Italian city would produce two thinkers that would focus a great deal of attention on what would seem to be on the surface nonsensical ideas, but that would likewise have long-lasting staying power in western thought. Both Parmenides and Zeno described the idea that motion did not exist, and they used the idea that the person would never reach a midpoint of their journey because the distance would grow smaller with each step, and the midpoint would constantly shift, thus there would be an infinite number of midpoints and the person would never reach the destination. Of course, we can show that we do reach the destination just by walking. The value of the idea is the fact that there are potentially infinite sizes of things, both big and small, and that this notion of the infinite allows us to conceive of things that are truly gigantic and things that are extremely diminutive (think the distance across the universe or the number of atoms in an object, or even the existence of the atom, for that matter). Both present many of their arguments as paradoxes, and it's the paradox that allowed philosophers and philosophy to sharpen their arguments.



Anaxagoras of Clazomenae
When we think of the Athenian persecution and the trial of Socrates where he pays with his life for the crime of impiety, Anaxagoras was another that became a target during that period. His views of the universe and his daring to utter that the sun was a ball of molten and fiery rock and not a god caused him to be the first of the philosophers of this time to be exiled from Athens. The main idea that Anaxagoras offers it the idea that all things are comprised of basic things and that nothing comes into being or perishes. This is akin to the law of conservation of matter, where nothing is ever truly created or destroyed, but rather transferred. Basic things contain a little part of everything, and these parts can be expanded or broken down infinitely. In Anaxagoras world view, there is no smallest thing, an item can be broken down ad infinatum and part of it would still be there. Finally, the mind is the action, the reason and the focus in which all things are put into play, and it is the mind that allows all matter to be formed and reformed. But it is the radical idea for its time that the moon and the planets were balls of rock in space that got him into the most hot water (saying that the moon's light was actually a reflection of the sun, and that the sun was a molten ball of fire). In hindsight, he was absolutely right! In others, he was very wrong, but the fact that he was able to focus on and come close in so many ways to the actual nature of the KOSMOS is interesting.



Empedocles of Acragas
When we think of a "Gandalf the Great" character to exist in the ancient world, in many ways, Empodocles fits the bill nicely. A flamboyant "Wizard" of a man, he nevertheless was an evangelical figure, one who believed strongly in the idea of saving souls and that he knew how to do it. The classical alchemical view of the four elements (Earth, Water, Air and Fire) and the two sources of all change (Love and Strife) are key to his philosophy.



Fifth Century Atomists: Leucippus and Democritus
If the idea of the atomic nature of things, that infinitesimally tiny items string together and become larger objects, and that all matter is comprised of them sounds familiar (and it should) then thank these two men and the Atomist school of thought that followed on with them. Though the idea was developed in this ancient time, that's not to say that it was wholly embraced, or that the atomic theories of Leucippus and Democritus are similar in all ways to our current understanding of them, but they definitely started the ball rolling with the idea and the philosophical reasoning behind it, that of the infinite number of building blocks existing within the void.


The book ends with a pair of chapters discussing the Sophists that immediately predated Socrates and the nature of the debate between Laws of Man and Laws of Nature, and the many participants and their ideas related to them.



Bottom Line:

Philosophy before Socrates is a dense book. It covers a lot of ground, numerous literary fragments, and a number of complimentary and conflicting systems and ways to look at the world and the understanding of knowledge. While many of the ideas and ideals espoused by these philosophers have been discredited today, it's the attempt that they made to rationally and through reason and debate come to understand their world that we see the seeds of Western thought, morals, logic, and understanding. To know where me might go, it's helpful to know where one has been, and this book definitely gives us that. Fans of philosophy will find this book fun in places and ponderous in others, but on the whole, it's an enjoyable read and helps to put into perspective many ideas that have formed and a looking glass to help bring them into better focus today. For those who are epistemology fans at heart, and like to know why we know what we know, this book is a gem.

When Catastrophic Failure Hits Home

I've wanted, and circumstances have kind of forced me, to take a break from the blog and contemplate on something that happened in my community over this past week. As many of you will see when you look at my profile, I live on the San Francisco Peninsula. More specifically, I live in the small town of San Bruno, CA. San Bruno became a headline news story last Friday and over the weekend all over the country. This was due to a natural gas line that exploded in a residential neighborhood. The explosion caused the confirmed fatalities for four people, destroyed several dozen homes, displaced hundreds of families, and required the concerted efforts of many cities fire crews and emergency personnel to maintain order and prevent the fire from spreading into other neighborhoods.

At 6:15 PM on Thursday evening, September 9, 2010, reports came in of a loud explosion in San Bruno near Crestmoor Canyon. Due to the proximity of San Bruno to the San Francisco International Airport, many thought that a jet airliner crashed in the canyon, and that the fire we were seeing was from such a crash. Shortly afterwards, we heard a report that all planes were accounted for, and that the odds of it being an airplane were non-existant. Then what? What was causing a massive fireball to shoot up in the air to heights of 100 feet and higher?

We didn't get to have the time to speculate on exactly what was happening, because our first motivation was to pack up our things and get out of the area as quickly as possible. Actually, this was more my wife and kids decision, because I had not returned home yet. I first learned of this situation when I got off of the Bay Area Rapid Transit (BART) train that I take every day to come home from work. As I drove up the street heading for my home, I saw the smoke and I saw the fireball, and it looked from my vantage point to be in the canyon. This scared me in a big way; Crestmoor Canyon is filled with Eucalyptus trees. Eucalyptus trees were what fueled the Berkeley Hills fire of 1991, in which 3000 homes and other buildings were destroyed. If Crestmoor Canyon were indeed on fire, and those Eucalyptus trees were to catch, then a large part of western San Bruno had the potential to go up in flames.

As I fought my way home through traffic and a police barricade and being told I could not drive into my neighborhood because an evacuation was in effect, I parked my car on the onramp to Southbound 280 and made my way by foot up to my wife's parent's place, where she and the kids were staying. From there, I was able to get the rest of the story. The fire was not in the canyon, but was in the neighborhood just to the west of the canyon, often referred to as "Crestmoor 2" by real estate agents, or Glen View/Claremont by two of the main streets. The cause was determined to be a 30 inch gas line that had ruptured, and it was the high pressure in this gas line that made the explosion so severe that it created a crater 40 feet across and 40 feet deep, while registering a 1.3 quake on the Richter scale as reported by the US Geological Service. The explosion took place very near the intersection of Glen View and Claremont, which is a low lying section of road locals call "the gulley" and where storm and water runoff heads into Crestmoor Canyon. The fireball that resulted from this explosion quickly incinerated the houses immediately surrounding it, and then the fire spread to other houses. In addition, the blast also damaged a water main, which meant there was no immediate water they could use to fight the fire; they had to either fill trucks some distance away or run hoses from hydrants more than 1/4 mile away.

Firefighters from several nearby cities were called in to help fight the blaze, work with the city and county to shut down the gas line that was feeding the fireball (and the fact that it had to be shut down slowly so as to not cause ruptures further down the pipe) plus a committed team of firefighters that were told, in no uncertain terms, "do whatever you have to do to make sure that fire does not hit the canyon!" They knew that, if the fire did spread to the canyon, many other neighborhoods would be in danger.

My family spent the night on the floor huddled around a battery powered radio (since the power was out for the greater western part of town), listening to the reports. With the smoke and the confusion, we were hearing estimates of 50 + houses destroyed and 120+ houses damaged. Knowing how many of our friends lived in that neighborhood, we knew this was not a good sign. With the power being out, we also had a hard time communicating with our friends and family to let them know we were safe. What I later found to be cool was the fact that a friend we did contact made a point to post to both my and my wife's Facebook pages that we were safe, had been evacuated, and that we were dealing with no power and an overloaded cell phone network and otherwise couldn't call or email to let people know we were OK. We later commented that this was the first "Facebook" enabled disaster, and that Facebook played a major role in helping get the word out to people about the event (expect me to do a post about that another time).

As the evening wore on, I decided that I needed to get my car off the freeway, and so I drove and found a place to park. As I was coming back up to where my wife's parents lived, I was stopped by the police and told I couldn't continue. I pulled out my wallet and said "look, I live here; my wife and children are just up the street. You may escort me up if you wish, you may arrest me if you wish, but one way or another, I'm getting back to my wife and kids!" With that, they nodded and let me pass and get back to my in-law's home.

Friday morning, we received the news that the evacuation order for our neighborhood had been removed, and that we could go back to our homes. We were overjoyed to find out that the fire had been contained to the Claremont area, that the fire did not make it down into the canyon and that our house was unaffected. With this brief joy, however, came the stark realization that many of our neighbors had lost everything. With power restored, we could see on the television and the Internet the pictures and video of the damage. Several dozen houses were gone. Not just burned down, but vaporized! Nothing left but chimneys and foundations. To this we received word that several people lost their lives, one of the fatalities being the older sister of a boy in the Cub Scout Pack that I used to lead, and that hundreds of families were displaced, for how long was anyone's guess.

As a software tester, I can't help but look at this in the lens of what I do for a living. I heard reports from people that said that due to a "glitch", the pipe ruptured. When a pipe ruptures and close to 50 houses are destroyed, and four people are confirmed dead, that is not a "glitch", that is a catastrophic failure! Many things have come to light from this incident that I had not known about my neighborhood, such as the fact that the Claremont area was sitting atop this 30 inch gas line running right through a densely populated neighborhood. I do not think any of the residents knew that. I certainly didn't, and I've lived here for 11 years. I did do some research to find out where the rest of these large scale pipes were located, and thankfully, the one that services my neighborhood is out by the main road that leads up to our neighborhood, but no houses are near it. Questions will certainly be asked, most of them pointed at Pacific Gas and Electric (P.G. & E.). During the reports, there were comments that residents' smelled gas for three weeks prior to the explosion, but little was known if anything was done about it. If indeed P.G. & E. did know there was an issue and did not do anything about it, then this rests squarely with them. Bigger questions will rise, such as "was the pipe decayed because of its location?" or "did running groundwater weaken the integrity of the pipe?", but I think one of the questions needs to be "who decided that building a densely populated community right over such a large gas main was a good idea?". Perhaps this is common, and I just wasn't aware of it before. Knowing what I know how, had I been in the market for a house, I'd think twice about buying a house where a primary 30 inch gas main lies just under my development.

Again, as a tester, I often think about these things in the abstract; a software failure that is deemed catastrophic is one where the system cannot perform its function. Even in such circumstances, the outcome rarely consists of more than lost time or loss of revenue (important in their own spheres, to be sure). In this case, a catastrophic failure has resulted in the deaths of four people (perhaps more), the destruction of dozens of homes and the displacement of hundreds of families. One thing is for certain, it's unlikely I'll be forgetting about this catastrophic failure any time soon!

TWiST #11: Twist Down Under!

This week's show was actually pretty fun to put together. First, my thanks to Farid Vaswani, one of our assistant producers, for helping get this interview together and for doing the first pass audio edit and leveling. Farid is based in New Zealand.

In this edition, Matt talks with Jared Quinert from Australia and Brian Osmond.  Jared has an eclectic background ranging from business software to video games to video poker!  Brian started out with the Inland Revenue Service of New Zealand and now trains testers at Software Education.

Anyway for those who want to check it out, here is Episode #11.


The podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to these whenever you want to :). Plus you can also get access to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.



Yeah, I'm shilling, I know, but STP hosts the podcasts and stores the archive, I think giving a little love back is perfectly appropriate under the circumstances, don't you :)?

No Book Review This Week

There's two reasons for this. First, the book I'm currently reading isn't finished, and I really don't like to review a book I haven't gotten all the way through, but that should be rectified by next week (for those interested, the book I'm currently reading is "Philosophy Before Socrates", written by Richard D. McKirahan).


The other reason I'm behind on my reading is that I'm spending a fair amount of my time reading the chapters of what will be the "Reducing the Cost of Testing" book that I am participating in. I'm reviewing the chapters currently and, based on what I'm seeing, I think this is going to be a really good book (of course, I'm a little biased due to a number of the contributors being testers I hold in high regard).


Anyway, look for my review of Philosophy Before Socrates next Wednesday.

A Tale of Two Workshops

This past weekend, my daughters and I went on our annual three day trip to Camp Pollock in Sacramento, hosted by CIHA (California Indian Hobbyist Association). This is the annual Labor Day Pow Wow weekend, which consists of a number of activities for families and friends that attend each year. This event, and others like it, has been happening for over 40 years, and some of the attendees have been coming for that amount of time. We came for our first time in 2007, so we're definitely the new family :). It's a great event that has a number of activities that celebrate Native American culture, dance, music, and craft. It's the last part of that list that had me musing about this today.

There were two primary craft-related workshops held over the weekend. One was based on the art of finger-weaving, which is a technique where yarn is twisted together to make cool patterns, and another was based on beadwork, where the project was to create a medallion. I sat with my girls during the finger-weaving workshop, and I stepped in to help my girls with their projects. My youngest daughter was having trouble getting started, so I offered to help get it going for her, and after I had completed several rows, I decided I could show her what to do, but as I turned to show her, she wasn't interested any more (at least not right then). I kept hoping that I'd get a chance to have her pick up where I left off, but again, she had decided there were other things she wanted to do.

For the beading class, since there was a cost for the materials, and since two workshops conflicted, and both daughters couldn't be in two places at one time, I decided to shadow both sessions, take pictures, and just watch. Since I couldn't actually sit in on the beading class, I had to let my older daughter just do the best she could and see if she could pick up the ideas and apply them. At the end of the class, she showed me what she had started, and that she was having some trouble in an area. Having been fresh off my experience with my younger daughter, I decided to tell her what I thought she should do, but I didn't touch the project at all; she had to physically do what I was suggesting. The net result was that she got past her sticking point and for the past couple of nights, she's been working on her project and getting farther and farther each day.

This reminded me a bit of some of the things I've done with co-workers and other testers. How often do we decide that we want to be helpful, but end up getting in and doing for the other person what they need to do to learn? It's so easy and so much more efficient to just get in and explain what we are doing as we do it. The only problem with that approach is that, while we may be able to help explain something and get some work accomplished quickly, the person who most needs to develop that skill gets short changed in the process, and in some cases, they may not ever approach the issue with the correct understanding, if they approach the issue again at all (they may well leave that to the "expert"). In the second approach, when we explain just what is needed but push and encourage the person to figure it out on their own, there is both an increase in understanding and an increase in the interest and focus to learning the task or skill, which tends to see an upswing in the interest in doing that skill again.

This week, I will take some time and sit down with my younger daughter, set up the original project, and let her practice with it from the start, so that she gets the hang of it, and can complete it on her own, without my help. Yes, she may get frustrated, and yes, she may decide it's not something she wants to do again, but she will get the chance to do it on her own, learn from her mistakes, and get the chance to complete the project on her own terms… and who knows, maybe she'll want to do more of it once she really gets how it's done.

End Note: the book at the top of the post today, for those who are interested, was written by Scott Sutton, who puts on the beading workshops regularly at these events. His work is fantastic, and he's an excellent teacher as well, so hey, check out his work :).

TWiST #10 Is Now Up: Performance Testing Pair Up

We had some interesting challenges with this week's podcast. This was the first time I was asked to sit in, and I even got to say a little bit at the shows intro, which was cool, but I basically let Matt and the guests do all the talking (LOL!). The audio could have been better, but I learned a lot about what top do and what not to do when it comes to multiple-parties in the same interview.





This week's show focuses on performance testing Dan Downing and Goranka Bjedov.  Goranka's an alumni of Google and now works with Facebook. Dan's background is in Insurance and Finance/Banking. As you can guess, those two environments have very different needs and requirements. Goranka and Dan are hosting a workshop on Performance Testing that will be offered at STPCon in Las Vegas on Monday, October 19^th, 2010, and talk a bit about it here.



Anyway for those who want to check it out, here is Episode #10.



The podcast is free for 30 days, but you have to be a basic member to access it. After 30 days, you have to have a Pro Membership to access it, so either head on over quickly (depending on when you see this) or consider upgrading to a Pro membership so that you can get to these whenever you want to :). Plus you can also get access to the entire archive of Software Test and Quality Assurance Magazine, and its issues under its former name, Software Test and Performance.

Wednesday Book Review: Web Security Testing Cookbook

Well, OK, actually, this is a Thursday Book Review. Yep, I'm late. Life intervened this week and I needed a little more time to complete this one.


Over the years, I used to joke about the fact that O'Reilly's books tended to be three types. The deep, reference books, the "hurry up and get through Chapter 3 so you can be productive" books, and the elusive 3rd category which is the "see how some stuff is done so you can learn to roll your own" books. Web Security Testing Cookbook, written by Paco Hope and Ben Walther, by its very name falls into the 3rd category; truthfully, these are the books I most appreciate by O'Reilly (I went back in my archives, and found it to be amazing that other than Beautiful Testing, this is the first O'Reilly colophon[1] book I've reviewed, which is shameful considering how many of their books I've owned).

One of the comments that used to be repeated by me and my co-workers over the years, and a strong plus in my opinion, was that the reason to get an O'Reilly book, if you read up to chapter three, you would know enough to be conversant and effective with the tool, application, OS, language, etc. being covered. In fact, it was rare that I ever read an O-Reilly colophon book cover to cover (it did, however, lend itself to the exasperating habit of making me an "advanced beginner" in many more subjects that I could have been much better versed in, but the titles were definitely much more well versed towards getting you in quick and getting on with things). The books that broke this trend were the Power Tools books, the Beautiful Books (see Beautiful Testing for example) and their "Cookbook "series. The benefits of the Cookbook is that each section has specific actions and specific items you can look at and consider for your use (and to use the cooking metaphor, many of us want to have an example recipe of Pad Thai to make ourselves rather than read about how amazing Pad Thai is or how to best enjoy Pad Thai).


Web Security is an interesting area, and often one that is traditionally examined at the end of the testing cycle. Not every test will be relevant to every tester, but there's something that will interest someone. The book starts out with an explanation of the Web based model and the various ways that web servers and clients can interact (from simple sites with a few scripts up to massive B2B sites managing huge business workflows). It then follows on with a series of free security testing tools. The biggest challenge with open source and free test tools is that the documentation is often sparse or non-existent (think about buying bulk spices; would you know how to use them in their current state?). Web Security Testing Cookbook lets the user get into the section that interests them, shows a tool that will help perform the task they are looking to accomplish, and put in place a sample script or application that will allow the user to consider other avenues and other scripts to create. The great thing about this book is that it is focused and it is brief (comes in at 320 pages). Each "Recipe" is structured in a similar manner. First a problem is displayed, then a solution is presented, and following the solution a discussion follows, so each recipe can be weighed and considered as to its usefulness. Each section begins with relatively simple projects and builds to bigger and more challenging scenarios.


Here's a breakdown of the chapters and topics:

Chapter 1: Introduction

This chapter introduced the user to the idea of performing web tests specific to security, and provides a rundown on the layers that are present with Web technology and how the various components within a web server and a web client actually work together. Most of all, this chapter explains that this book is a "HOW" book, and

Chapter 2: Installing Some Free Tools

A number of tools ranging from Firefox and Firefox plug-ins like Firebug, installing Perl and Perl packages, cygwin (unix/linux functionality for Windows), plus a number of other tools that will help with specific testing options.

Chapter 3: Basic Observation

This section goes through a number of basic scenarios ranging from viewing the HTML source to examining request headers and live post data, seeing hidden fields and detecting JavaScript events

Chapter 4: Web-Oriented Data Encoding

This section shows the user how to examine various data encoding schemes such as Base 64, Base 36, URL encoding, etc, not so much as a reference, but to get the user familiar with the format and recognize it when they see it. The point is also plainly made that encoding is NOT the same thing as encryption; these encoded fields can be very easily converted back to plain text with a minimum of effort.

Chapter 5: Tampering with Input

The best way to see how a web page or application responds to malicious input is, well, to send it malicious input. This chapter discusses methods and ways to do exactly that. This section shows ways to muck around with forms, files, GET and POST commands, manipulating cookies, and other ways to mess with the data that is directed to the server (including uploading sample virus files).

Chapter 6: Automated Bulk Scanning
This section deals with tools to map, or "spider" your web application by following and making a reference to every link and every input on the page. In addition to spidering the section approached mirroring and scanning sites.


Chapter 7: Automating Specific Tasks with cURL

cURL is a command line tool that allows testers to automate specific tasks. This section goes through a number of methods to fetch pages, follow redirects, look for cross-site scripting and impersonating a particular web browser, and then gets into some more sophisticated options that can be used such asmanipulating cookies or creating multi-stage tests.

Chapter 8: Automating with LibWWWPerl

Perl has long been a mainstay in Web programming; most of the CGI scripts from the past two decades were/are written in Perl, and many enhancements that allow for relatively easy scripting of web transactions. Perl's LWP library allows the user the ability to perform various security tests. Examples range from basic fetching of pages, simulating form input, sending malicious cookie values, uploading malicious file contents, uploading viruses, and editing a page directly through a script.

Chapter 9: Seeking Design Flaws
There is a difference between a bug and a design flaw. Bugs are generally localized issues with intended code, and can often be fixed fairly easily. Design flaws are situations where a user can take advantage of a system or use a system in ways the designer or developer never intended or hadn't even considered. The recipes in this section help the user identify various situations such as bypassing pages that restrict the users access (or not), examining password recovery programs, manipulating lookup information to find other people's details in a database, and running repeated attacks (Denial of Service, etc.).

Chapter 10: Attacking AJAX

AJAX is one of the cornerstones of the new interactive WEB 2.0 model, where instead of the user explicitly calling for changes, the application can do it by itself at timed intervals or in slam localized areas based on user input. Note: this book doesn't get into the details of AJAX specifically, but it does show how to use a number of tools, such ad Firebug, to view the actual requests, interpret and modify requests, intercept and modify server responses, and injecting various data types to change and modify AJAX transactions.

Chapter 11: Manipulating Sessions

This chapter shows the user how to find session identifiers in cookies and requests, how to parse and understand authorization headers, and change session details to evade restrictions, gain privileges and impersonate other users.

Chapter 12: Multifaceted Tests

The last chapter assumes that the user has read through and tried the various attacks and recipes in the previous chapter. If each section previous was seen as cooking an appetizer or an entrée (hey, it's a "cookbook" so forgive the oft repeated analogy), then Chapter 12 is the equivalent of preparing Thanksgiving Dinner (or any other big festival or celebration to those not specifically from the U.S.A.). The purposes of this chapter is to tie many of the previous examples together into longer and more detailed attack strategies.



Bottom Line:

This is not a book you read cover to cover. It's a book you sample, and as time and opportunity provide, you try out various recipes, add a little or take away a little here and there, until you have something you can add to your repertoire, then you move onto another one, and another, until you have an arsenal of good tools and the ability to blend them together as needed. Some time will be needed to get used to some of the methods, and some time will need to be invested into using the tools demonstrated. The more attention and time paid to these tools, the sooner you can move from being a web security "cook" and join the ranks of the web security "chef's". Unfortunately, no one can write a "chef's" book, but I'll dare say this particular cookbook will get you a goodly distance towards that goal.






[1] Col-o-phon, n., a publisher's or printer's distinctive emblem, used as an identifying device on its books and other works. When I say a "Colophon" book, I'm referring to the distinct to O'Reilly design that it has a wood block print of an animal on the cover, which has been a stylistic element of O'Reilly books since their inception. In my day to day work, I rarely refer to an O'Reilly title by its name, but by its animal, such as "Hey, can I borrow the Turtle book (Korn Shell)?" or "Who has the Chimpanzee book (Exploring Expect)?" or "Hey, where's my Llama Book (Programming Perl)?" The funny thing is, most of us (meaning me and my co-workers) who regularly read O'Reilly titles know exactly which book we are referring to just by saying the animal on the cover. That's some pretty awesome iconography, if nothing else. For those wondering what you would say for this one, my guess is you'll be asking "Hey, who has the Nutcracker book?!", since the Nutcracker is the bird on the cover.