10 Years on a Rocket Ride: A Meandering Walk Through Twenty Years of Software Testing

For the first three years of working with Cisco Systems, I had a peripheral relationship to software testing. It was something that I did, but it wasn't my primary responsibility. Mostly, I focused on maintaining a lab to keep the people in Release Engineering happy, and as far as I was concerned, for the most part, that meant working for and helping make sure that the infrastructure to test builds was working, and so that customer problems could be investigated.


I knew who the software development engineers were. They were the ones who wrote the actual code that went into the images. I definitely knew who the micro-coders were, as they were the ones writing the code that went on the interface boards. I knew who the software engineers were that worked on the release engineering side, but I didn't make the connection that they were software testers until considerably later. Why? Well, first off, there was nothing in their title that led me to believe they were testers. All of them were "Software Engineers". The "Testers", i.e. those who were officially titled as such, all worked with hardware, and for the most part, they were responsible for rework of the PCB's and making sure they worked.


I think for me, this idea of being a tester has been bred into me from years of hearing about testing of physical products, and was driven home by the old Hanes Underwear commercials and the "Inspected by #12" commercials. I know I'm really showing my age by that reference, but what I mean to say is that I looked at the group doing testing as the group that was physically testing the assembly line, those physically inspecting boards and electronics. I didn't look at the idea of their being a difference between software development and software test until I met a new hire at the time named "Chuck". Chuck was the first guy on the team to clue me in to the fact that what he did was software testing. I met Chuck because he was one of the Release Engineering team new hires who was willing to roll up his sleeves and help with designing the next hardware testing lab to go online. Because of this, we worked closely together, and as we talked more, I got to understand his role in the company and why he was hired. His phrase has been a hallmark of mine for almost 20 years... "I was hired because I like to break things!"


This was the first time I heard about this kind of approach. I had heard of the testers checking to make sure that boards worked, but the thought of actually deliberately being destructive hadn't occurred to me. One of the ways that this was manifested (and very clearly) was when one of the Release Engineers wanted to get some error output for debugging purposes, but they weren't sure how to do it. Chuck smiled, pulled out a couple of paper clips, and went over to where an AUI cable was plugged in (yes, I remember when Ethernet was too thick to fit into an RJ-45 connector :) ), and jammed the paper clips into the AUI Connector. He then walked over to the console window and lo and behold, errors!


This was a whole new idea to me, that software testing could be, well, negative, and aggressive, and have a touch of "weird science" to it. Because of this, and the fact that Chuck and I had a lot in common outside of work (music, video games, spicy food, etc.) I got to spend a lot of time talking with Chuck, and Chuck introduced me to the world of software testing. It was in this world I started to understand that Release Engineering was more than build engineers, that every software engineer that worked for release engineering but wasn't specifically a build engineer was in fact a software tester. Technical software testers, engineering grade software testers, able to code in C and C++ software testers, but still software testers. Through Chuck I got to see more of how the bug tracking system worked and the back and forth between the software developers and the release engineering group (of testers) and what I saw surprised me. There was a bit of a rivalry going on there, and there was a bit of back and forth with each bug. Chuck taught me a number of idioms that would be familiar to many testers; throwing over the wall, the dividing line, us vs. them, breaking things for fun and profit, and the all encompassing "wow, that's interesting... can you do it again?"


As Cisco kept growing exponentially each year, more and more testers were joining the team. Around 1993, an initiative was undertaken to help Cisco manage its immense growth, and at the same time, work to secure contracts in the broader marketplace, especially Europe. This initiative was called ISO 9001, and it touched on every part of the organization. As part of this change., the original Release Engineering group was split into three area. The first was the build group, and it would in a sense stand alone from this point going forward. The second group was Development Test, and the third was Product Test. This was the first time that Test as a group in the software realm was codified and made explicit, and it was around this time that I had to decide which group I would support (actually, it wasn't so much "make a choice" as it was that we had three administrators, and it made sense to have a point of contact for each group. Since Chuck was now part of Development Test, I decided to be the point of contact for that group, and with it, I started to work more closely with a core team that was now specifically labeled as Development Test and people who now, for the first time, were called "Dev-Test Engineers".


As the growth kept escalating within Cisco, we still had the problem of too much work and not enough people, no matter how fast we hired, so a lot of the manual testing tasks were offered to me, and I accepted them. Over time, a number of the Dev Test team (which was now swelling with the ranks of Manufacturing "Testers" who made the switch over to Engineering) and I felt that it made sense to have me be more directly involved with this group. As had happened a couple of years before, this DevTest crew went to bat for me, and asked if I could be moved into the group as a Dev Test Engineer. The manager for the Dev Test team liked what I was doing, and made it happen. Remember, I'm a kid with no tech training, no CS or EE degree, and no experience with "testing" outside of what I'd picked up from this group. Still, the crew that was there felt I'd be able to do a good job on the team, so in 1994, I was re-assigned to the Development Test group as a junior Dev Test Engineer.


In many ways, the following six years would be opportunities for me to learn more about testing from the Cisco perspective, to work with various groups and responsibilities (Enhanced Online Insertion and Removal, Performance Testing, Stress Testing, Negative Testing, and a bit of Automated Testing using the Tcl/Tk language and toolkit). I have to admit I was never really good on the programming end, but I did strive to get better at it. Cisco supported testers and engineers in getting further education, and through that support I went and completed a UNIX Programming Certificate through UC Santa Cruz. I also went back to Mission College in Santa Clara during these years to focus on and develop some better Computer Science chops. To that end, I decided it was time to bite the bullet and dive into an area that I was woefully deficient in, and that was mathematics. I went in and took classes starting with first semester College Algebra, and worked my way through over the next several semesters through  Pre-Calculus and Trigonometry (about two years worth of total classes), until I completed first semester Calculus with a B. Make no mistake, I considered that one of the greatest academic achievements of my life at that point :). I also did a number of "Direct Study" projects because what I was working on at Cisco was more advanced than what I would be able to take classes for at Mission. This two year period helped me greatly in filling in a lot of general education requirements that I hadn't taken in my five year stint at DVC so many years earlier. Cisco also sponsored several training brown bags and official product trainings, and I attended a bunch of those, but there was very little in the way of official tester training offered. We tended to fend for ourselves and collaborate together to improve our craft.


During the six years that I was directly testing, I worked on big teams that were pushing out full platforms like the 7500 and 7200 routers, the Catalyst 5000 switch, and the PIX firewall, as well as in much smaller groups (or even by myself) on projects like the CiscoWorks Network Management platform, and the NetFlow FlowCollector & FlowAnalyzer. I finished up my tenure at Cisco working with the Content Management group and working on the Cisco Cache Engine and Content Switching team.


One of the things I can say for certain about the Cisco experience was the focus on documentation. Test plans, project plans, test scripts, etc. were the order of the day, and the more detailed, the better. I cannot count how many times I went int to present test plans and then being told that there was "not enough detail" in the plans. Looking back, I really wish I had the presence of mind to say "detail for what purpose?!" But alas, I wasn't at that level of focus or understanding.


There is no denying the fact that the growth of both the physical company and the growth in their stock price had a big effect on my staying as long as I did. It had its ups and downs, but through the 90s it always came back bigger and more valuable than ever. Cisco's stock had jumped so much in value that it allowed in, in one fell swoop, to exercise options, sell off and pay the taxes necessary, and net enough cash to pay off all of my debts by the middle of 1995. Ten years of debt management came to an end, and with it, I could now sell off my option shares and bank the rest, which I did religiously for the next five years. In 1999, my family was now me, my wife and two young children and with the shares I had saved, I had enough to sell and buy a house with. I bought most of it up front, but took a small mortgage because I didn't want to have to sell ALL of my share to buy the house. To say that Cisco's stock had been very good to me in the 90s would be a tremendous understatement.


Of course, no good thing goes on forever, or as a Chinese proverb well puts it "no tree grows to Heaven". In 2000, I saw the signs of change coming; many of the perks were being restrained for the company, one which had been enjoying a screaming growth curve for a decade. My commute for years had been long and, well, hellish, but it was worth it while the stock was rocketing upward. As it started to come down back to earth, and the value of my shares were coming more in line with a new reality (and for the first time, future shares were well underwater) I started to question whether or not it made sense to keep commuting so long and being away from my family for so many hours. The Golden Handcuffs were less appealing, and I was getting offers from other places to go and see if I'd be a good fit for them. One of these companies was Connectix, located in San Mateo (just ten miles from my house) and the thought of being able to do what I did closer to home really appealed to me. Thus, after ten years, I decided it was time to get off the rocket ride and bid farewell to Cisco. A lot of the old timers were sad to see me go, and I remember well a number of people who told me that "I helped make this Company", and that meant a lot to me.


So what did I learn from this time and interaction?

- I learned that there was a distinction between software development and software testing, regardless of the labels applied and the titles used. Developers are developers, and testers are testers, and while the organization can try to foster a different culture, the groups know the difference.


- I had some great interactions with development managers and individual developers that helped me make my bugs more focused and meaningful, and help me look at what I was really trying to solve. I also learned that a priority to me was not necessarily a priority to development. I also learned that having a high "Junked" bug count was the worst indictment a tester could have (whether this was meant to be the case or not, I did whatever it took to make sure my bugs would not be junked).


- I discovered that a lot of projects made sense to have a test team, and others made sense to have just one tester. As a Lone Tester, there were skills I had to step up and develop because I couldn't rely on other testers to fill in the cracks.


- The World Wide Web was the best friend of a tester if you knew where to look. three cheers for dejanews (look it up for those of you who weren't there ;); it was basically the webified version of NNTP newsgroups).


- Cem Kaner's book "Testing Computer Software" would become a good friend, but alas, one I wished I'd read more carefully and completely. Still, with the areas I did read, I did well.


Coming from Cisco, I had the world at my feet and the ability to do just about anything I wanted to do. In a boom economy, you can do things like that. In a bust economy, though, one quickly discovers one may not be as good or as talented as they have led themselves to believe. Needless to say, 2002 would give me my first understanding of this... but I'm getting ahead of myself ;).