Wednesday Book Review: How We Test Software at Microsoft (Afterward)

After so much time focusing on just this title, I figured a summary review was finally in order, so here it is, my overall review of How We Test Software at Microsoft.


This has been a long and sometimes challenging journey. When I set out to first do this project, I had a specific goal in mind, and I figured that I would apply this to my company and see what elements would be useful. Little did I know that, at the time I agreed to do this, that by the time I finished it, I would be operating in a totally different sphere, at a totally different company, in a totally different development environment.


First, a refresher on how this project came about. I was at the Pacific Northwest Software Quality Conference in October of 2010, and I met Alan Page as he was giving a talk on performing code reviews from the perspective of being a tester. At the end of the talk he was giving this book away to those who asked questions and participated. As he had the last copy he was giving out, I impetuously said “Hey Alan, if you give me a copy I promise to write a review about it!” He smiled and said “OK, you’re on” and with that, my copy of HWTSAM fell into my hands.


As I thought about how I wanted to do this, I remembered an approach that I thought was really interesting was done by Trent Hamm over at The Simple Dollar, which is a personal finance blog I’ve followed for several years. Trent is a voracious reader, and he would often do as many as two financial book reviews a week! Now, to be fair and provide some context and contrast, Trent is a professional blogger, The Simple Dollar is his primary job and source of livelihood, so he has the time to focus on that level of reading. TESTHEAD, while important to me and a tool I regularly use to help sharpen my craft as a tester, is not an income generator, nor is it meant to be. Likewise, it’s hard to work full time, raise a family and work on a blog to try to give it the attention it deserves. Two book reviews a week would be out of the question, and this book would be too dense to do as a one week review and a general list-out. Then I saw Trent’s Book Club entries and I said “A-ha! Now that’s an approach I can work with”. A chapter every few days shouldn’t be a problem!

Shortly after I started reviewing this book, my reality took a dramatic change. I was offered the chance to start a test group at a startup company, and I took it. Those who follow this blog know that's how I transitioned from Tracker Corp. to Sidereel and in the process, decided to move on from a company that I’d been with for six years. Needless to say, the challenge of managing that transition caused the updates to slow from twice a week to once a week, and then for a stretch I was only able to do two chapters over a three week period. I didn’t want to drag this out for too long, so I put in a huge push to get the last few chapters all ready to publish within a couple of days.

So what did I learn from this process? Was this approach a good one? From a Book Club standpoint, I think it went terrible, as I didn’t receive a single comment for any of the chapters. However, looking back over the stats of the site, many of them were in the top twenty posts on the site while I was publishing the chapter commentaries, so I know they were being read :).

From a learning and a pondering standpoint, I thought the process was great. I had the chance to spend a lot of time mulling over the ideas in the book, and rather than offer up a review or summary of each chapter (I'll direct the reader to the individual chapter synopses I've already provided). I liked some chapters more than others, and I’ll confess Ken’s chapter on Services was really dense and took me quite a bit of time to work through. Overall, I thought the book was very helpful and interesting. Some people are critical about the book as being too specific to Microsoft, but I found the approach to be interesting in an anthropological sense. I felt like the curtain was being pulled back and I could see what was actually happening inside of Microsoft. More to the point I could see where many of the tactics and approaches were similar to what I would do, and I could see processes and steps that were totally foreign to my approach and way of thinking. Was I happy that I had a chance to learn about these approaches and philosophy? Yes! Will I use every one of the methods described? Heck, no! Will I use even half of what is described in this book? Likely not, but even if I only walk away with half a dozen new ideas to consider, that’s a great percentage.

To be clear, I do not nor have I ever worked for Microsoft. I’ve worked with a Microsoft partner (Connectix) that was ultimately acquired by and became part of Microsoft, so I can say that I know several people that now work for Microsoft, but I don’t have any requirement to provide a glowing review. In some parts the book was excellent, and in some it was slow going and borderline not practical for what I do. Still, even with those criticisms out of the way, there’s a lot of solid information in this book. From understanding the role of test in the organization, to creating test cases, understanding the different product lines and methodologies used to test, to setting up and utilizing virtual machines, there’s lot of ground covered in this book. While some would criticize the lack of specific details, that’s not really the point of the book. The authors expect a far amount of testing knowledge coming in; i.e. this is not a book for beginners. Having said that, there’s still plenty of information; even a beginner could feel productive and find some ideas to help “bump up their game”.

This is an experiential book, and it’s based on the real world lessons of three Microsoft veterans. It’s heavy on Microsoft specific jargon, but as the book progresses, that jargon becomes less and less of an issue, and by the end of the book, even obscure acronyms don’t seem out of place, because you now have enough context to see what they might be called, and 8 out of 10 times, you’d be right. If there is going to be a criticism, it will of course have to be that the Microsoft centric focus is going to be less than an optimal transplant to another environment, especially those that are not running with Microsoft technologies such as .NET.


There is a fair amount of bureaucracy in the pages, but that is to be expected for a company with so many testers; a common language and methodology was chosen and agreed to so as to help facilitate communication. While Microsoft will never be confused with a 20 person agile shop, it certainly has groups that are nimble and approach development with more in common to the small agile shop than to the behemoth standards of yesteryear.


So having now spent so much time on this project, and others associated with my job transition, I can give some ideas as to what I would do differently were I to approach this BOOK CLUB method again.

1. Doing a book club review process and setting up a Practicum process for another book and running them at the same time proved to be a lot more than I could easily chew and digest, especially when I was trying to post more than one chapter a week. With hindsight, I would run these reviews at two different times. I don’t regret the approach for either, I just wish I’d done them at separate times so I could devote my whole attention to each one of them.

2. Each chapter took about three hours to read, ponder and write up. Sometimes that was relatively easy, and at other times, those three hours were just impossible to find! Were I to do it again, I would commit to a single chapter each week and write the review in parallel with each section and sub-section. That would stretch out a sixteen chapter book to nearly four months, but it would allow the reader to really spend some time with each chapter and cover it extensively. For the record, I completed all 16 chapters in 11 weeks, and that was with posting the last three chapters three days apart from each other. A more even spacing would have probably felt more natural and would have also given me some more time to absorb the material even more.

So this officially draws to a close the emphasis I’ve placed on How We Test Software at Microsoft. My thanks to Alan for giving me the book to review. I don’t think that this was quite what you had in mind when I said I’d review it, but I hope it was worth the time and the wait. Happy Testing!