Learning Ropes and Making Tradeoffs: A Meandering Walk Through Twenty Years of Software Testing

When I found myself working at Cisco Systems in 1991, I found myself working with a fascinating mix of very eclectic people. Release Engineering was the group that I would be working with specifically, but because I was actually burning images into EEPROMS, I was actually considered a contractor working for Manufacturing. Manufacturing was in one of the three buildings and when we say manufacturing, we really mean assembly. We sourced various parts from other companies and we had a group of people that actually assembled them. In addition to programming EEPROMS on both an individual level and on a much larger level with seriously big machines that could burn 8 sets of EEPROMS at a time (64 total), I felt like I was living in the world of The Jetsons.


I also had the chance during these early days to get two different visions of testing. In the Manufacturing side of the company, testing was synonymous with hardware rework. Everyone would go through and work at their stations checking boards using oscilloscopes and determining if the electronic pathways were working right. If they weren't they were able to drop in resistors and wires and fix the problems. these were the classic "testers", so in many ways, I felt like that wasn't something I'd be capable of doing. In Release Engineering, though, there was much less hardware rework and much more talk about protocols and interfaces and throttling and other things that has less to do with physical hardware tweaking and much more to do with playing around with the console and sending commands.

An early, and exciting, interaction came when I met my good friend Shannah, who I still communicate with to this day :). Shannah was a release engineer, and she also spent a lot of her time managing the "testing lab". At first, I thought that this was going to be another place where people reworked boards, but when I came in, I was surprised to see that it was racks of equipment and a lot of cables branching out to various computers. Words like Ethernet, Token Ring, FDDI and X.25 were used, and I saw a bank of big white square boxes that, from my time working in the manufacturing area here and there, I realized was the flagship product of Cisco, the AGS+.  As I looked around at the various patch panels and the systems that were on both sides of the room (Sun SPARC 2 workstations, a couple of VAX/VMS servers, various HP-UX systems, plus several variations of PCs and Macintosh systems, I made a mental jump that would help me immensely... this place looks like a recording studio! Not in the sense of the mixing board and tape machines, but in the mass of cables going from machine to machine and having patch panels there to help you route the signals where you wanted to have them go. Now THAT I could understand!

Over the next several months I would do what they needed to have me do so that I could get images burned and released, and to help me also keep track of what was going on, I had my little DEC terminal replaced with what was to me at the time one of the coolest things I'd ever seen. I was given a 21" NCD, which was effectively a graphical UNIX terminal. It allowed me to use X Windows and spawn terminal windows on several machines, give me access to various tools that let me see how I could access this wild thing called Email, and most amazing of all (to me), use a tool called XRN that allowed me to access this insanely wild world of information called "newsgroups".

Another fact that I was not entirely aware of, but would come to discover soon enough, was that Cisco was on  a growth curve that was, to put it quite simply, insane. From the time that I worked there, from March 1991 until February 2001, the company would grow by 100% or more each year, and sometime by much more than that. The net result was that there was a lot of work that had to be done and not enough people to do it. There was no question that I was not at the time qualified in the slightest to be interviewed for a permanent position with the company, but since I  came in as a temp, I could easily be applied to any and all "grunt work" jobs that others either didn't have the time for or found less than tasteful to do. As for me, I was just happy to be able to work steadily, and for more money than I'd ever made up to that point. The staggering sum? $8.00 per hour, plus overtime if I worked it, and double time if I worked it. For several months, I really had the ability to do as much as I could stand, and I could be rewarded nicely for doing it (nice is relative here, remember; as a musician, outside of gigs, I'd made maybe a little better than minimum wage).

The time on the EEPROM programmer was limited; I could only burn one set of images at a time reliably, and since they were connected by a serial connection, it meant that an average image would take a half an hour to burn. This gave me a lot of time to go and help Shannah with the build out of the testing lab in our building, and then to do more build outs for several more labs that were to come on line within the next several months. Setting up the racks, building the PC's, loading the racks, and stringing all of the cables through the wire ladders was actually a lot of fun. It was physical work, and it was very tangible. When a lab was ready, it was obvious how much went into to putting it together. Of course, all of this equipment had to be maintained, and had to be "tested" to make sure that everything was running as it should. To that end, I started to learn how to "telnet" to the various systems console ports and issue commands from my NCD terminal. It felt so high tech and space age. Later I'd realize that, outside of the Cisco specific routers and terminal servers, I was working with what was considered at the time "low end gear", but to me, I felt like I'd been given a shot at being some kind of "mad scientist".

After ten months, several lab build outs, lots of cable wiring, and more beta software images than I could count, I had made friends with a number of the release engineering team, and they all went to bat for me to get me hired on full time. They felt that I'd done a really good job keeping the labs in shape, and that if I were to be hired on full time, I should be the junior Lab Administrator, along with some others that would be hired to help manage the workload. They realized there was more to do than I could reasonably do even if I worked 14 hours a day (and some days, I  did).

The year that was my break through into the tech world was also a year that was to "crater" the music scene that I was a part of. All this time, I was making money, paying off debt, financing a new but very modest car, and putting money towards getting to that next level. 1991 was known for many things, but in the music industry, it was known as "the year Nirvana changed everything". Truth be told, Nirvana wasn't the change so much as the over-saturation of hair metal ballads was. I'll make no bones about it, I loved being a glam performer. I loved being outrageous, being larger than life, putting on a show, and looking somewhat off the wall. In 1991, al that changed. If you were still wearing make-up and big hair then, you were a relic. If you sang about good times and played sleazy grooves, you were also a relic. Grunge was the order of the day, and it was a time of adapt or die. Industrial was also becoming more mainstream with ultra heavy beat that you could dance to as well as rock out to, and truthfully, this appealed to me more than Grunge did.

My band morphed into a semi combination of Grunge and Industrial, where I deliberately lowered my voice to less of a high tenor and more of a high baritone. It sounded forced, and while musically it was interesting, I was having a lot less fun doing it. Line-up changes were inevitable, people came and went, and finally, in the late summer of 1992, my band mates decided that they wanted to move to Los Angeles and seek the record deal and go for it all. For the first time in ten years, I found myself in a strange situation. First, I found myself in a situation where I actually had something to lose. Second, would I want to:
-  go and drop everything
- lose all of what I'd learned and understood
- to go to a market where my new found skills didn't apply
- and start all over with the debt load that I had (which had actually increased with the money I was making)
- so I could maybe get a record deal in a scene I didn't recognize anymore and play a music I wasn't really enthusiastic about playing
- and lose the connection with the fans that actually supported us all these years in the area where we'd made our mark, i.e. San Francisco?

Even with all of that, it was the most agonizing decision I had ever made up to that point in my life. How do you effectively drop nearly ten years of your life?! Was this company I was working for really all that? It took me some time to come to some simple conclusions:

1. I was in debt up to my eyeballs, and were I to go down to LA, I would have to carry all of that with me.

2. The scene I was part of had changed, and I was enjoying it a lot less than I had two years previously.

3. In just the 18 months I'd worked at Cisco Systems, I'd seen the company grow from around 300 people to close to 1,200 people, with new offices opening up on the east coast and other places.

4. I started to pay attention to a little number called "stock options" and ESPP shares, and looked at the stock price. It was doubling every year, and the stocks themselves were splitting just about every year. I did the math and realized that, if the current trend held, I could conceivably work my way out of debt in two years. I had no idea if I'd ever be able to do that if I left and went to LA, even if we did get the big record deal. We'd had a chance to see many of our friends who did get record deals and we saw the truth about their financial situations. They were dire. Yes, they'd had some fame, but they were also broke and deeply in debt. I decided I just didn't want to live like that any longer.

Thus it was in those late summer months of 1992 that I decided it was time to make a change, and let the band I'd worked on for so many years go their own way, and I wished them the best of luck. From there, I decided it was time to put my head into overdrive and see what this tech world really was all about. I would play with other bands over the next couple of years, but really and truly, that magic period where I was "all in" for music had ended, and the prospects of getting married, having a family and exploring some other goals in life took center stage.

So what had I learned during these couple of years?

- If you are lucky enough to find a company that has more work than what it can reasonably get done, don't be afraid to volunteer for grunt details. Sure, it's not glamorous work, but in many ways, you can learn a lot about the systems from a holistic standpoint, and get to see what all the moving parts can do.

- Even the best and most accomplished engineers don't know everything about the systems they work with. You can be a valuable source of information about what the product is doing in  a significantly complex arrangement.

- We had a gathering every week with our customers that came to our training sessions called "Beer and Cookies" and I'd walk around and get to know our customers and see what they were doing, where they worked, and what they were experiencing. Again, because of my years as a musician, I'd learned how to go about and chat with people and do some "promotion". I decided to use some of those skills in this setting, but rather than promote Cisco (let's face it, they were already customers, there was no point in my doing that) I listened to what problems they were having (and there were many of them, to be sure). I would then try to go back to the labs we had and see if I could set something up similar to what they were describing and see if I could make it do the same thing.

- Release Engineering was, for all practical purposes, the group that built the software to be released to the public. It was also the software testing group. It took me a while to figure that out, but figure it out I did. Most of the people I was working for and with, the ones who were my strongest allies in helping me get hired, were testers.

I began to self identify as a software tester as well, since I was supporting and helping them do their work. In a short period of time, I would be given the opportunity to see what being a tester, for real and for full time, was all about. I'd also see what being strapped to a rocket ride for 10 years can do to a person, for good and, perhaps, not so good.