BOOK CLUB: How We Test Software at Microsoft (1/16)

Since I started this blog, one of the positives I’ve heard back from readers about is the fact that I would do a weekly book review. I would go into some depth on why I think it’s worthwhile, usually with a chapter by chapter breakdown.


This approach to book reviews is not mine. I actually first saw it done this way by Trent Hamm over at The Simple Dollar, and liked the format so much I adopted the approach for TESTHEAD, right down to trying to do a review each week.


Some of you may notice that I’ve fallen off the wagon with the weekly book reviews. There are a few reasons. The first is that I’ve already gone through most of the books that I have that are related to testing, and the second is that there are a number of books that, while I’d like to review and cover, deserve more than a week to read, digest and comment on.


This brings me to Trent and the Simple Dollar again. Occasionally, he does a multi-post book review in more of a “reader’s club” format, where his review is not of the entire book, but of each chapter, spread out over several weeks. I’ve decided to do exactly this, to give others a chance to consider the chapters, but more to the point explain what I felt was of value regarding the chapters and what my takeaways were, in a way that a quick five sentence summary paragraph just can’t do.


Alan Page made the decision to feature the first book in this format easy when he gave me a copy of “How We Test Software at Microsoft” (Microsoft Press, 12/2008, 978-0-7356-2425-2), co-written with Ken Johnson and B.J. Rollinson. Since I promised Alan I’d give him a good and thorough review, I felt there was no better book to start the “TESTHEAD BOOK CLUB” than this one :).


Introduction:


Alan tells us why he decided to write this book and why he felt that another book about software testing was valuable. In addition, he describes the training that Microsoft gives its testers and makes a case that it’s the format of the training, with its stories, anecdotes and odd pieces of trivia that make it memorable and ultimately effective for its participants. In short, while the information is helpful, it’s the stories that are embedded with the techie stuff that makes it worthwhile.


In How We Test Software at Microsoft, Alan has separated the book into four themes. The first deals with Microsoft in general, and what makes it an interesting eco-system in regards to its practices with people and engineering. From there, a section would be dedicated to the actual methods of testing employed by Microsoft. The third section would cover the tools that are used to help make that testing a reality. The final section deals with what Alan and the authors see as the future of testing at Microsoft.


Alan makes the point that this book is not just his, and that Ken and B.J. have a significant impact on the content in the book. Each of their styles comes through in the book, and they all make their mark. Alan described B.J. as being the “professor” of the bunch. Ken is the “historian and storyteller”. Alan is the Agent Joe Friday of the group with the “just the fact, maam” attitude.


Alan also makes it clear from the outset that this book cannot possibly take every aspect of testing at a company the size of Microsoft and do it justice, not can he claim that everything in the book is to the letter what testers at Microsoft do all the time, but it represents a good cross section of the company and the way that they approach software testing, the methodologies they use, how they are applied, and what they feel works for them.


In addition, there is a web site specific to book and commentary related to it at http://www.hwtsam.com/, and there are, at the time of my writing this, 45 entries, so feel free to check the site out if you’d like to read more about the book and commentary related to it from the source.


Also, so as to keep everyone on the same page, there will be times where I’m going on what the book is saying, but it may not be the latest and greatest reality (it’s been two years since the book was published, so note in my review when I use the parenthetical “(AOP)” it means “As Of Publishing”).


Chapter 1: Software Engineering at Microsoft


This is a brief overview on the way that Software Engineering is conducted at Microsoft, and some of the key aspects that help to make Microsoft culture what it is. Microsoft has gone through changes in its approach and goals, starting with the vision of  “a PC on every desk and in every home”, and then moving on to “Empowering people through great software – any time, any place, and on any device”. In 2002, the vision shifted "To enable people and businesses throughout the world to realize their full potential." At the time of the writing of the book, Microsoft’s vision statement was "Create experiences that combine the magic of software with the power of Internet services across a world of devices." This goes well beyond the original idea of a PC on every desk, and it shows Microsoft’s willingness to adapt to a changing marketplace (and for that matter, the necessity to do so).


I’ve only worked for one really large company in my career, (my definition of really large meaning a company with more that 10,000 people in it), and that would be Cisco Systems, though some could argue that Konami fits the bill as well with over 5000. YMMV. The idea of working for a company as immense as Microsoft honestly boggles my mind. There are three main product divisions in Microsoft (AOP):

• PSD Group (Platforms and Solutions)
• MDB Group (Software for Businesses)
• E & D Group (Entertainment)

Each of these divisions is reported to earn $10 billion to $20 billion in revenue, making each of these divisions larger that many Fortune 500 companies!


Alan makes the first of the diversion in the book to tell a story about T-shirts and the joke that “when a T-shirt has been made, it’s a sign that reorganization is in the works”. Having had the big company experience with Cisco Systems in the 1990’s, I can very much relate to this story. It also put a smile on my face and proved Alan’s point in the introduction… five years from now, I’m probably not going to remember the details of Microsoft’s organizational structure, but I’ll smile when I think of the T-shirt story, because it reminds me of my own experiences.


Think about the sheer numbers at Microsoft… they employ more than 90,000 people worldwide (AOP), and the point made about Microsoft’s size is their product breadth. More to the point, this breadth has unique engineering challenges that need to be met. Microsoft has applications, games, peripherals and devices in just about every conceivable software market, ranging from Datacenter Server Clustering software to Halo 3. Key to this is the understanding that there is no one way to build and ship products. Relevant to the tester, that also means that there is no one way to test their products, either. There is a set of guiding principles and ideas that they use, some broad enough to cover all product lines, some specific enough that they are only relevant to their local application.


Irada Sadyknhova shares an anecdote where shipping a product with Microsoft is a lot like producing a play. There are directors, producers, actors (the example likens the engineers to the actors), and while a performance of a play is about the art of the play, the end goal is to sell tickets to their performances. Likewise, Microsoft orchestrates large scale shipping product launches with one key goal in mind, to find an audience willing to use, become loyal to and pay for the experience.


“The key to the whole analogy is balancing the contradictions of a large software company that has to produce big profits with high margins and be dynamic and creative at the very same time. There is some inherent conflict between large production scale and creativity; balancing them successfully is a core of the success of Microsoft. No Google, or Apple, or Sun has quite yet mastered this challenge at scale. Only the likes of Cirque du Soleil and Microsoft have proven they can do it.”
—Irada Sadykhova (Director, Engineering Learning & Organization Effectiveness – Engineering Excellence)


Microsoft also champions the ideas of looking for that next new market or idea, realizing that not all of them are going to pan out. Many times, product enhancements that we see in the final version started out as incubation projects, often in very small groups as just an idea, and allowed to develop and grow in influence as the application or approach demonstrated that it had “legs” to stand on. The Bill Gates Think Week is another example, where White papers are presented and Bill Gates reads and comments on hem twice each year. Often these papers become incubation projects as well.


Microsoft employs 10 different types of engineering disciplines. They are as follows:

• Software Development Engineers in Test (SDETs) are responsible for maintaining the testing and QA standards.
• Software Development Engineers (SDEs) are the coders, i.e. the traditional Software developners that write the code that ultimately becomes Microsoft product.
• Program Management (PM) combines project management, product planning, and design together. The PM defines new product’s technical details and helps ensure it gets made.
• Operations (Ops) manages and maintains Microsoft online services and internal IT details.
• Usability and Design Usability Experience and Design (UX) focus on the visual end-user experience. Conducts research to see how the user interacts with the product, and then proposes improvements based on the feedback.
• Content creates the UI text, articles, training guides, magazine and web articles, books, Help files, etc.
• Creative describes positions most often associated with the Games group, working on new ideas for games and game play for PC and Xbox titles.
• Research focuses on publishing papers, and helping to allow new technologies to incubate and form.
• Localization International Project Engineering (IPE) focuses on translating from English to multiple languages (and from multiple languages to English, too), as well as adapting software to local market requirements and needs.
• Engineering Management run the teams that employ the other engineering disciplines.

Microsoft has development centers spread out all over the world, with a bulk of their engineering taking place in the U.S. (about 73% in and around Redmond, Washington, USA), but also including development offices in places like India, Ireland, United Kingdom, Denmark, Japan, China and Israel. They predict that a larger part of that development work will shift to other geographical areas, and that shift will increase in the coming years.


Coming up on Thursday, I will cover Chapter #2, which is about Software Test Engineers at Microsoft. Stay tuned until then :).