Not All Fun and Games: A Meandering Walk Through Twenty Years of Software Testing

It's June of 2003. I had resigned myself to focusing on school, and I figured that would be the bulk of my time going forward, and there would be little in the way of my having a realistic amount of time to actually work simultaneously, but there was that strange Craigslist ad... a QA tester that had an excellent singing voice? What was that all about?


I didn’t have to wait very long to find out. The ad was placed by Konami of America, located in Redwood City at the time, and it was a call for testers to help them with their “rhythm games” offerings. Konami is known for a number of different businesses around the world, but their most famous is their video games division, and specifically titles like Dance Dance Revolution. In 2003, Konami was working with Boston game developer Harmonix to publish a game called Karaoke Revolution, a singing game that tested your vocal ability and scored you based on accuracy and timing. The problem? None of the testers were able to test the game at the Expert mode level. For this purpose, they needed to hire testers that could, well, really sing.


I went in to interview and I discovered a sub-culture of the testing world I was, up until that time, totally unfamiliar with, that of the “temporary game tester”. For those who don't know about this model, here’s how it existed in 2003. Testers were contracted through various temp agencies, and depending on the company, you were ether hired by the project or for a set contract time. Often the standard block of time was either six months or one year. During that contract, you were paid the industry standard (in 2003, that standard was $12 an hour). Very often, there were periods where overtime was expected (sometimes lots of overtime) to make sure that a game could meet a release date. At the end of the contract, many workers would find that they were automatically let go and had a non-hire clause of 90 days (yep, that meant that the company that hired you would let you go at the end of the contract and they wouldn’t hire you again for 90 days after your previous contract). For this purpose, a lot of game testers were nomadic, and worked for a lot of different publishers, and frequently moved from company to company to keep steady work.


This was a world I was totally unfamiliar with in my testing experiences, but since my focus was on getting through school, any work I could find to help me slow down the burn rate of my savings was welcome, and the fact that I had savings to burn though helped me to say “well, this could be an interesting experience, what have I got to lose?” Shortly after the interview, I was offered a contract and I took it. Interestingly, my first week was not actually testing Karaoke Revolution, but working on a PC game that had a lot of chapters, a multi-player ability and a need for a lot of video card and sound card testing with various combinations of components. Thus, on June 30th, 2003, I found myself in a big room with a bunch of other game testers, many of them long time veterans of this scene, and me, starting over again.


A little bit about game testing. Lots of people look at it like it’s a dream job. “Oh, how awesome, you get paid to play video games all day, what a life!” Well, that may be true for the first couple of days, but really, there’s a big difference between playing a video game and testing a video game. Playing a game means walking through a level and enjoying the scenery and interacting with the story and “riding the rails of the title”. Testing a game means taking a character and running them along the walls, solid objects and other characters and making sure they don’t fall through the cracks. It means going through the same dialogs over and over to make sure that they are displayed correctly. It means walking areas over and over and over again after making small hardware changes such as a different video card, sound card, OS or memory change. In short, actually playing the game is the smallest part of the job, and often, you don’t even get an awesome title to test. For every Metal Gear Solid, there is a Barbie’s Horse Adventure or a Rainbow Bright Studio, and yes, I know that those games exist (LOL!).


One of the things that made testing with Konami interesting was that there were two types of projects. The first were games published in the states primarily and made with US development houses. Those games were a little easier to communicate through, and we used a standard bug tracking system and communicated back and forth between the developers and testers fairly easily. The biggest challenge was that builds would be shipped to us on DVD and we’d need to install them and test them from physical media that had been shipped. This meant that our testing was often gauged in a week’s worth of time, and the turnaround for any given cycle was often a week, sometimes more.


The second testing area was the titles Konami directly developed, usually in Japan, and were sent to us in the US to test and make sure they were ready for release into the U.S. and English speaking markets. This was a very different system, and one that required precise language and wording for both bugs and status reports. Every day, I would have to submit a testing report and bug entries, and they had to be reviewed by the Senior Tester for the QA group. The reason for this was to make sure that there was no ambiguity for our translator. Yep, our reports would be manually translated from English to Japanese and sent over to Japan for review and further comment, and then we’d get the follow-up in our morning meeting each day.


I had the chance to lead the Karaoke Revolution project with a number of other testers and work on a range of testing challenges. In addition to the singing aspects, there were the “Technical Resource Checks” that were required for any title that would be published by Sony (for the Playstation series), by Microsoft (for the Xbox) and Nintendo (for any of their consoles) and these checklists had to be adhered to exactly. Either they passed or they didn’t. Another aspect of this world that was very strange was the “lock boxes” for the consoles and the PC’s. The workstations had literal cages surrounding them, and locks were applied to them. If you needed to change media or make a component change, the senior tester would then come in and unlock the box so you could make the change, and then lock it up again. Truth be told, I found this practice to be demeaning, but I also understood that the game industry has a history of theft and black market piracy, especially for unreleased games, so I also understood the need for the precautions.


After the Karaoke Revolution game was shipped, a number of us were reassigned to help out with the Customer Service group. This was a common practice since the Customer Service needs often spiked around various titles releases, and rather than hire and lay off Customer Service reps, they kept a small group of CS people and added or diminished from their ranks by using testers. I got assigned to the CS group and dug into the then massive list of emails, letters and phone calls that we were getting. Over time, I started to notice that the other testers were put on various testing projects, but several months later, I was one of the last of the testers to not be on a new project. I asked the Senior Tester if I’d done something wrong, and his answer was “actually, no, you did something really right. Not to belittle our testers, but for many of the games, well, we can pick and choose anyone to test or lead those titles, but it takes a real talent to effectively calm down angry customers and help them resolve their issues. You’re not still doing Customer Service because you are being punished. You are still doing Customer Service because you are remarkably good at it!”


For much of my time at Konami, I would straddle these two worlds. I would use my testing skills to help answer customer problems and investigate issues and devise workarounds. In these days, unless something was truly catastrophic, a bug that was shipped was eternal. We didn’t fix the systems, we figured out how to get around a problem area and then published it. Thus, I got a lot of experience in doing “forensic testing” or testing products that had already been released and had known issues, and my job was to find ways to get people “unstuck” where possible and help them keep playing and get past the pain point. That’s a very different kind of testing as compared to saying “this area doesn’t look right, it needs to be redesigned”.


Games are a very cyclical market, with a lot of build up and intensity in the spring, summer and fall months, and a huge dip during the holiday season (everything needs to be ready to be purchased in time for the holidays, so most work is wrapped up two to three months prior). This means it’s common for layoffs to happen during this down time, and I was no exception. Granted I was prepared for it by the nature of the work and how it was presented, but a part of me was a little sad about it. While there was a fair amount of turnover, there was a tight camaraderie among the testers at Konami, and each time someone was let go, it felt like we were losing a friend, and many felt the same way when I was told it was my turn. I was OK with it, though, as I had plenty of other thing vying for my attention, such as a full time school load.


I mentioned that I was doing school through an online program through the University of Phoenix. I think it only fair I give you an honest review of that approach. I have heard both good and ill spoken of the UoP system of education, and I will say that for my purposes, it actually worked pretty well for me. I cash flowed my entire time there, and didn’t borrow any money whatsoever. I was also fortunate in that my Dad helped us out, too. He said that, out of all his kids, I had been the “least expensive” when it came to college education. He compared what he had spent helping my siblings get through school over the years and based on that average, offered to split the tuition costs with me. Interestingly, I discovered that, because of my “large” amount of stock equity, I was totally ineligible for any college grants or financial aid, which made sense, I guess ;). Still, when you are writing a check for $2000 for each class, that has a galvanizing effect to make sure that you did well in those classes, regardless of what the requirements, other students or external community might be saying. For me, I put everything I had into it, and I minimized the time between classes by doing a double rolling schedule. This meant that every class that was scheduled for four weeks would be over-layed on top of another class, but staggered by two weeks. This way, I was doing the peak work for one class while another on was ending or starting. Also, with the strict 9:00 am – 5:30 pm work schedule at Konami, plus other things I was doing with my family, I discovered the prime time for me to study was from 4:30 am to 8:30 am each day. I also discovered that, for all intents and purposes, that was the ideal time for me to actually be studying. I came to the realization at this time that my “brain waves”, for some weird reason, were most in tune to actual learning and problem solving at these ridiculous hours.


My reprieve to focus entirely on school would only last about a month, though. Konami called me and asked if I’d be willing to come back and do another contract. It seems that while I was gone, the Customer Service needs had skyrocketed, and the CS team, when asked if they needed help, actually said “yeah, can we get Michael back?” For the record, I thought that was rather cool, and since I still had well over a years worth of classes to get through, I said sure. Over the coming year, I would cycle on and off on projects related to testing titles. I’d also test three more versions of the Karaoke Revolution series (the reason I was hired in the first place) and in a neat twist, I’d get to lend my voice to the game. For those interested, if you play Karaoke Revolution and you load the Doobie Brothers song “China Grove”, like all songs, you have a choice of having the guide vocal in the mix to sing along with, or you can mute it and just sing by yourself. The guide vocal for “China Grove”… is me :).


As 2004 wound down, I realized that my time in school was also winding down. In the space of 20 months, I would complete 22 classes for a total of 66 units, and I’d maintain a GPA of 3.96 through that process. As I was now getting to the point where I had to make a decision work wise, I did state to Konami that I might be willing to work permanently with them if they were to make me a full time job offer, but if that were not to materialize, I would be looking elsewhere in earnest. My completion date for my degree was February 25th, 2005. The preceding three months, I’d interviewed for more testing jobs, and while the market had changed a bit in those ensuing two years, it was still the same laundry list of “we’d love to hire you as a tester, but really, we’re looking for junior level software developers that we can grow into developers, and we’re putting them into testing positions for half the salary of seasoned developers”.


Gah!!!


Well, I’d just spent the better part of two years doing Customer Service and Technical Support, what does the job market look like for those positions? Actually, I was surprised to find that there were lots of them! Based on my recent classes and my degree completion, I found that, when I submitted my resume for tech support and customer service jobs, I was getting a call-back ratio of two to one over my resume being submitted for testing positions. I also decided to do something different with my resume this time around. Instead of sending out a bullet point resume with keywords and job descriptions and responsibilities, I made an essay version of my resume, telling about what I learned at each job and how I applied it. This proved to be very effective, in that I received many more call backs with this type of resume than with the standard bullet point approach. I also started using a new tool called LinkedIn, and set up my profile and started to add friends and contacts to it. While this was in the very early days of LinkedIn, I found that it gave me a chance to compare what I was presenting and modify it regularly. I also found that having people I worked with endorse me also added a little bit of a boost to my visibility for job opportunities.


After several interviews, I finally accepted a job with ImmigrationTracker (a company that would later change its name to Tracker Corp.)… as a Customer Support Specialist. Wait, what?! But your career is testing, you all would say. Why would you do that?! Well, truth be told, I had seen what the state of testing was like in the middle of the decade of the 2000s in the Bay Area, and I was not very encouraged. I also saw that people actually wanted my problem solving skills in the Customer Service area. I figured “hey, maybe it’s time to put these testing skills to use in another endeavor” and really, I figured that, maybe, my testing days were over and I was now working in an area that was a better fit for me personally.



Ah, but don’t you fret, dear reader, as you already know... there’s a few twists and turns still to be had. This blog title is called TESTHEAD after all, and I talk about software testing, not customer service and technical support, so you would be wise to think that my days as full time tech support would be limited… but that, as they say, is a different story, and I’ll tell that one to you tomorrow :).


So, did I learn something new from these experiences? Oh goodness, yes!!!


- - Was the degree worth it? Financially, I would say it was a wash. Knowing what I know now and the opportunities I have been able to take advantage of in just the past three years, had I done then what I have done more recently, it’s entirely possible that I would have had a greater impact doing TESTHEAD back in 2003 rather than focus on a bachelors degree. Having said that, I did learn a great deal about myself; my capacity to work, study, reason and discover new things that I might not have been able to do had I not gone back to school. So while I may question the financial aspects of going back to school, I do not question at all the personal changes and discoveries I made about myself and my abilities that I learned while I was doing it.


- - Game testing is hard work, and it suffers a bit from the same stigma that the music industry has. Because so many people want to be a part of it, there’s a huge and competitive group of people doing this kind of work. It’s brutal, and it’s really not designed to be for everyone. If you survive in the game industry, it’s because you really want to be a part of the game industry. Many people toil for years before they get a break to become a designer or a producer or have some other role in the industry. Some even stay in the testing realm and work their way up, but make no mistake, these people work very hard to get where they are, and their effort is very commendable. Yes, many people come into the industry, but only a handful really have the fortitude to stick it out for the long haul. While I like video games, I decided I didn't like them that much!


- - Customer Service is really another side of the testing coin, and if you have solid testing skills, you may find that you also have the ability to do really well supporting users and helping them through their pain points. It was also a plus that my “rock star” voice and my time spent schmoozing all those years in the music scene helped me to work with a lot of disgruntled people and help “talk them off the ledge” and get to a point where we either resolved the problem, or at least found a way to get around their issue and move on.


- - If you want to know if you are communicating clearly, try writing all of your bugs knowing that they will be translated into Japanese. It’s an eye opener how much of our everyday vernacular creeps into our writing (yes, even writing bugs) and upon review, we realize that “we keep using those words. I don’t think it means what you think it means!”. Much of this approach I still carry with me to this day. I still write my bugs as though I am anticipating them being translated for someone else to read.


- - Sometimes it’s the little things that drive people crazy. The big show stopper bugs get the press, but it’s the little areas of a game’s movement, the repetitiveness of dialog, or the really trivial aspects of an area’s design that makes people get really irked. While we may think an area is too small to be considered, or it’s so trifling that a customer would never complain about it, I have lots of phone messages and emails that state otherwise. Again, you never really know which issues will set someone off.