#CAST2015: Full Program is Posted

It's that time of year again. The time when the #CAST2015 hash tag starts to get a lot of play. For those who stumbled here from another location, #CAST2015 refers to the Conference for the Association for Software Testing, which has been an event that, for my not being directly involved in its planning or execution, has taken up a fair number of cycles of my reality. 

The full program was announced today, and it is available to view here.

To paraphrase the Conference Committee (or more accurately, just steal it whole cloth):

The Association for Software Testing is pleased to announce its tenth annual conference, CAST 2015 “Moving Testing Forward,”to be held in Grand Rapids, Michigan, August 3-5.Since our first CAST we  have seen dramatic changes in the nature of communications and the nature of delivery, from PC to client/server to the web and web services. Deployment is different; monitoring is different, builds and test tooling are different. We have a variety of new models and methods for our testing. CAST is where we talk about how they actually work out in practice, based on experience. At our tenth CAST, in 2015 speakers will be presenting stories, workshops and tutorials regarding their experiences surrounding how to advance software testing.

Join us this summer for our tenth annual conference in downtown Grand Rapids at the beautiful Grand Plaza Hotel August 3-5, as we explore “Moving Testing Forward.”

As President of the Association for Software Testing, you can bet I will be addressing a whole lot of CAST stuff over that the official web site, but right now, I'm actually stepping back a little bit and I'm talking totally off the cuff, not as the President of AST (though this will certainly be seen by some as "official pronouncements" anyway, so whatever).

First, I want to say that I am impressed with the variety and variation we have in the program this year. CAST is a diverse conference to begin with, but we made a specific focus this year to invite people who had not presented before to really give it a go this year. I am happy to say that there are several speakers who will be giving their first conference talks at CAST this year. I'm also proud that AST partnered with Speak Easy and arranged to work with them to help develop talks for the conference. Speak Easy has made it its mission to help encourage women to speak at conferences, and we are delighted to say that four talks that were mentored by SpeakEasy were picked for the conference.  CAST has prided itself on being a place where different voices get heard, and not specifically catering to the "rock stars" of our industry. We've had a good balance, I think between male and female speakers, male and female keynotes, and a diverse group of participants from different countries and backgrounds. Compared to many conferences I've been to, I'll dare say I think CAST really is one that deserves high marks for diversity. Can we do better? Most certainly, but this year's lineup makes me feel like we are in the vanguard. 

I'll borrow a couple of posts from Lisa Crispin to help make this point even more ;):

Another area I want to talk about is the two hour workshop that will be offered on Tuesday, August 4, 2015 called "Black Box Accessibility Testing: A Heuristic Approach". This workshop will be given by Albert Gareev, with some solid peanut gallery support from yours truly. I'm sure some of you are looking at the title and thinking to yourself "Wait a minute, that sounds a lot like the talks Michael has been delivering this year. Are they at all related?" The answer is, of course, yes. Albert and I have been working for quite some time to help develop both design and testing approaches that help bring Accessibility to light and get some focus and attention. We've both written and presented extensively on these topics, and this year, we have melded minds to help present and deliver a solid workshop of testing skills and techniques that anyone can walk away with and be effective. So why is Albert listed as the speaker and not me? First and foremost reason, the workshop is primarily Albert's research, practice and learnings, so he very much deserves the credit for presenting the workshop. Do I have a hand in what's being presented? Sure, and I'll be spending a fair amount of time helping hammer out the paper that will be available at the conference (yes, we are writing the paper together :) ).

For those who want to attend a conference that will be first rate with regard to content, diverse speakers, unique and original voices, and talk about topics that are relevant to your everyday work, and not retreads of stuff you've heard many times before, I want to personally encourage you to sign up and attend #CAST2015 with us in Grand Rapids, Michigan. It's going to be a great time, and I hope to see you there as we work to "Move Testing Forward"! 

 

When The Music's Over

The past two and a half weeks have been very difficult for me to process. Part of me is numb, part of me is frustrated, and part of me is deeply sad. All of these feelings have conspired to render my writing nearly non-existant, because I couldn't produce anything until I addressed the overwhelming elephant in my reality.

On Friday, April 17, 2015, Socialtext Director of Quality Assurance, Kenneth Pier, completed his battle with cancer. We received word around noon that he had passed away at home, surrounded by family. Two and a half weeks earlier, we spoke on the phone together to make sure I knew the steps to perform a release from start to finish. At the end of that conversation, he told me he felt very tired, and might need to take a day or two off. I answered "by all means, do what you need to do. We'll talk again when you feel better". That was the last time we'd speak to each other directly.


I don't want to talk about losing Ken. Instead, I want to talk about how he lived, and what he meant to me. I met Ken for the first time in December, 2010. Matt Heusser invited me to a "tester's dinner" at Jing Jing's in Palo Alto. The resulting conversations turned into TWIST podcast #30. I remember being impressed with Ken right off the bat. He was a little gruff, and no-nonsense with his answers, but he was a straight shooter, and he possessed a wealth of skill and a practical approach that he was happy to share.


Over the following two years, I would run into Ken at various meetups, workshops and conferences. Each time, we'd talk more in depth, getting to know each other better. In the summer of 2012, I had the chance to sit in with him and Cem Kaner at CAST as they discussed ways to balance automation and exploration. As I was set to give a talk in two days on the same subject, that interaction and deep questioning caused me to discard much of my original talk and start fresh. Ken took the time to answer dozens of questions I had, and in the process, helped me make a talk I would deliver several times over the ensuing years. That final product was hugely inspired by Ken.


A few months later, when I expressed an interest in a change of scenery, and a desire to work with a broader testing team rather than keep going it alone, Ken lobbied me to join his team at Socialtext. I accepted, and thus began a daily interaction with him that lasted for two and a half years. I learned a great deal from Ken and his unique style. Ken was not one to idly chat or play games. If he felt you were doing something good, he told you, immediately. If he felt you were losing focus or going off the rails, he told you... immediately :)! You always knew exactly where you stood with him, what was working for him, and what wasn't. He also took great care in representing his team to upper management, both within Socialtext itself, and when we were acquired by PeopleFluent. Ken was fearless when it came to telling people what could be done and what couldn't. He didn't care if upper management was frustrated or irritated with an answer, he'd rather give them the straight truth than make a promise we couldn't deliver, or push something that would be sub-par.


During many of the stand-up meetings we'd have each morning, Ken would have his oldies station playing over the phone system (half our team is distributed). Some mornings, he'd start singing at the start of stand-up, and often, he'd ask me to sing along with him, since we were both vocal performers (he sang with a number of choir groups, and had a wonderful baritone voice). Over time, though, due to the cancer treatments, the singing voice was quieted, and we heard it less and less. Now, the singing has stopped. I won't hear it again in my day to day work life any longer. I think I will miss that most of all.


I want to invite my friends who are wine connoisseurs (because Ken was definitely one of those) to raise a glass to Ken. He was a man who understood testing, and represented that understanding very well. He inspired a lot of love and admiration from his team, and from everyone that knew him. He was generous with his time, energy, and knowledge. He loved to roll up his sleeves and do the day to day work that, by virtue of his position, he absolutely did not have to do. Still, more than just doing the day to day work, he reveled in helping his team learn something new, do something they'd never done before, and encourage them to go farther and get better each and every day. It's a trait I hope I can exhibit when I work with others in the future.


Thank you, Ken, for just plain being amazing. Our team has a massive hole in it, and I doubt we will ever truly fill it, but we will do our level best to make you proud of us nonetheless.

On Community, or "Building Your Perfect Beast"

This is going to come across as perhaps a bit scatter-shot, because it contains thoughts that address a lot of things that I am involved in. Each of these could be addressed in a number of different places, and in time will very likely be addressed there, but for right now, today, I am going to separate my various public personas and address this as just me, right now, In my own sphere. What you are reading right now has both nothing to do and everything to do with the Association for Software Testing, the Miagi-do School of Software Testing, Weekend Testing Americas, the Bay Area Software Testing Meetup group, and the freelance entity that is TESTHEAD. For right now, I am addressing this as me and only me. The remarks that follow are not necessarily representative of any of those other initiatives and the other people involved in them.

One of the key messages I received loud and clear about the world of software testing and those who are involved in it is that there is a bit of an identity crisis happening. The world of software development is changing. Long standing biases, beliefs and processes are being disrupted regularly. Things that worked for years are not working so well any longer. Business models are being upended. The rules of the game and the very game itself is being re-written, and many people are waiting to be shown the way.

One of the things I have witnessed time and time again is that there is no end of people happy to consume materials created. I'm very much one of those people, and I enjoy doing so. However, at some point, there comes to be a space where there is a limit to what can be consumed and a void as to what is there. Something is lacking, and a need that is not being fulfilled. It's very easy to ask that someone fill that need, and complain when that person or group does not do so. I remember very well being called to task about this very thing. Back in 2010, I lamented that no one had brought Weekend testing to the Americas, that I had to attend sessions hosted in India, Europe and Australia, often either late at night or early in the morning for me. Joe Strazzere set me straight very quickly when he said (I'm paraphrasing):

"Perhaps it's because no one in the USA values it enough to make it happen. YOU obviously don't value it enough either. If you did, you would have already made it happen!"

That was a slap, but it was an astute and accurate slap. I was lamenting the fact that something I wanted to consume wasn't readily available for me in a way that I wanted to have it. I was waiting for someone else to create it, so I could come along and feed at the counter. Instead, I realized someone had to prepare the food, and since I wanted to eat it, I might as well prepare it, too. The rest, as they say, is history, and to be continued.

I want to make a suggestion to those out there who see a need, an empty space, an unfulfilled yearning for something that you have identified... is there any reason why you are not doing something to bring it to life? Are you waiting for someone else to give you permission? Are you afraid your idea may be laughed at or criticized? If you are a software tester, are you suffering from the malady that you see a problem for every solution? If so, I want to assure you that I understand completely, as I have been there and find myself in that position frequently. Perhaps you feel that you are alone in your frustration, that you are the only one who finds it frustrating that something you care about is not being addressed. While I was at STP-CON, and during Kate Falanga's session, we discussed the three layers of engagement and change/influence. The first layer is ourselves, the second is our team or organization, and the third is the broader community. There's a very good chance that any void you are seeing, any initiative that you hope to see championed, has many who likewise want to see a champion emerge.

My advice to all out there is to stop waiting for someone else to Build the Perfect Beast, and instead, start building your own. Once you start, it's a good bet others will want to get involved as well. No one wanted to do Weekend Testing in the Americas until I was willing to throw my hat in the ring. In short order, others decided they wanted to get involved as well. Some have come and gone, but we are still here and many have helped us solidify what we do. Our Beast is not yet perfect, but it's getting there, and we've learned much along the way. Same goes for every other organization I am involved in. Major movements do not happen by timidly waiting for someone else to take the lead, and they don't come about by asking for permission to do them, either. If you see a need that is not being met, try to create something that will meet that need, even if you have to do it alone at first. My guess is, over time, others will see what you are doing and want to be part of it, too. Do be warned, though, the desire to create is addicting, and you could find yourself a bit over-extended. On the other hand, you may be having too much fun to care :).

Delivering The Goods: A Live Blog from #STPCON, Spring 2015

Two days goes by very fast when you are live-blogging each session. It's already Thursday, and at least for me, the conference will end at 5:00 p.m. today, followed by a return to the airport and a flight back home. Much gets packed into these couple of days, and many of the more interesting conversations we have had have been follow-ups outside of the sessions, including a fun discussion that happened during dinner with a number of the participants (sorry, no live blog of that, unless you count the tweet where I am lamenting a comparison of testers to hipsters ;) ). I'll include a couple of after hour shots just to show that it's not all work and conferring at these things:

---

Today I am going to try an experiment. I have a good idea the sessions I want to attend, and this way, I can give you an idea of what I will be covering. Again, some of these may matter to you, some may not. At least this way, at the end of each of these sessions, you will know if you want to tune in to see what I say (and this way I can give fair warning to everyone that I will do my best to keep my shotgun typing to a minimum. I should also say thank you (I think ;) ) to those who ran with the mimi-meme of my comment yesterday with hashtag "TOO LOUD" (LOL!).

---

9:00 am - 10:00 am
KEYNOTE: THINKING FAST AND SLOW – FOR TESTERS’ EVERYDAY LIFE
Joseph Ours

Joseph based his talk on the Daniel Kahneman book  "Thinking Fast and Slow". The premise of the book is that we have two fundamental thinking systems. The first thinking system is the "fast" one, where we can do things rapidly and with little need for extended thought. It's instinctive. By contrast, there's another thinking approach that's required that makes us slow down to work through the steps. that is our non-instinctual thinking, it requires deeper thought and more time. both of these approaches are necessary, but there's a cost to switch between the two. It helps to illustrate how making that jump can lose us time, productivity and focus. I appreciate this acutely, because I do struggle with context-switching in my own reality.

One of the tools I use if I have to deal with an interruption is that I ask myself if I'm willing to lose four hours to take care of it. Does that sound extreme? Maybe, but it helps me really appreciate what happens when I am willing to jump out of flow. By scheduling things in four hour blocks, or even two hour blocks, I can make sure that I don't lose more time than I intend to. Even good and positive interruptions can kill productivity because of this context switch (jumping out of testing to go sit in a meeting for a story kickoff). Sure,the meeting may have only been fifteen minutes, but getting back into my testing flow might take forty five minutes or more to get back to that optimal focus again.

Joseph used a few examples to illustrate the times when certain things were likely to happen or be more likely to be effective (I've played with this quite a bit over the years, so I'll chime in my agreement or or disagreement.

• When is the best time to convince someone to change their mind?

This was an exercise where we saw words that represented colors, and we needed to call out the words based on a selected set of rules. When there was just one color to substitute with a different word, it was easier to follow along. When there were more words to substitute, it went much slower and it was harder to make that substitution. In this we found our natural resistance to change our mind as to what we are perceiving. The reason we did better than other groups who tested this was that our ability to work through the exercise was much more likely to be successful in the morning after breakfast rather than later in the day after we are a little fatigued. Meal breaks tend to allow us to change our opinions or minds because blood sugar gives us energy to consider other options. If we are low on blood sugar, the odds of persuading a different view are much lower.

• How do you optimize your tasks and schedule?

Is there a best time for creativity? I know a bit of this, as I've written on it before, so spoilers, there are times, but they vary from person to person. Generally speaking, there are two waves that people ride throughout the day, and the way that we see things is dependent on these waves. I've found for myself that the thorniest problems and the writing I like to do I can get done early in the morning (read this as early early, like 4 or 5 am) and around 2:00 p.m. I have always used this to think that these are my most creative times... and actually, that's not quite accurate. What I am actually doing is using my most focused and critical thinking time to accomplish creative tasks. That's not the same thing as when I am actually able to "be creative". What I am likely doing is actually putting to output the processing I've done on the creative ideas I've considered. When did I consider those ideas? Probably at the times when my critical thinking is at a low. I've often said this is the time I dod my busywork because I can't really be creative. Ironically, the "busy work time" is likely when I start to form creative ideas, but I don't have that "oh, wow, this is great, strike now" moment until those critical thinking peaks. What's cool is that these ideas do make sense. By chunking time around tasks that are optimized for critical thinking peaks and scheduling busy work for down periods, I'm making some room for creative thought.

Does silence work for or against you?

Sometimes when we are silent when people speak, we may create a tension that causes people to react in a variety of different ways. I offered to Joseph that silence as a received communication from a listener back to me tends to make me talk more. This can be good, or it can cause me to give away more than I intend to. The challenge is that silence doesn't necessarily mean that they disagree, are mad, or are aloof. They may just genuinely be thinking, withholding comment, or perhaps they are showing they don't have an opinion. The key is that silence is a tool, and sometimes, it can work in unique and interesting ways. As a recipient, it lets you reflect. as a speaker, it can draw people out. the trick is to be willing to use it, in both directions.

---

10:15 am - 11:15 am
RISK VS COST: UNDERSTANDING AND APPLYING A RISK BASED TEST MODEL
Jeff Porter

In an ideal world, we have plenty of time, plenty of people, and plenty of system resources, and assisting tools to do everything we need to do. Problem is, there's no such thing as that ideal environment, especially today. We have pressures to release more often, sometimes daily. While Agile methodologies encourage us to slice super thin, the fact is, we still have the same pressures and realities. Instead of shipping a major release once or twice a year, we ship a feature or a fix each day. The time needs are still the same, and the fact is, there is not enough time, money, system resources or people to do everything comprehensively, at least not in a way that would be economically feasible.

Since we can't guarantee completeness in any of these categories, there are genuine risks to releasing anything. we operate at a distinct advantage if we do not acknowledge and understand this. As software testers, we may or may not be the ones to do a risk assessment, but we absolutely need to be part of the process, and we need to be asking questions about the risks of any given project. Once we have identified what the risks are, we can prioritize them, we can identify them, and from that, we can start considering how to address them or mitigate them.

Scope of a project will define risk. User base will affect risk. Time to market is a specific risk. User sentiment may become a risk. Comparable products behaving in a fundamentally different manner than what we believe our product should do is also a risk. We can mess this up royally if we are not careful.

In the real world, complete and comprehensive testing is not possible for any product. That means that you will always leave things untested. It's inevitable. By definition, there's a risk you will miss something important, and leave yourself open to the Joe Strazzere Admonition ("Perhaps they should have tested that more!").

Test plans can be used effectively, not as a laundry list of what we will do, but as a definition and declaration of our risks, with prescriptive ideas as to how we will test to mitigate those risks. With the push to removal of wasteful documentation, I think this would be very helpful. Lists of test cases that may or may not be run aren't very helpful, but developing charters based on risks identified? That's useful and not wasteful documentation. In addition, have conversations with the programmers and fellow testers. Get to understand their challenges and areas that are causing them consternation. It's a good bet that if they tell you a particular area has been giving them trouble, or has taken more time than they expected, that's a good indication that you have a risk area to test.

It's tempting to think that we can automate much of these interactions, but the risk assessment, mitigation, analysis and game plan development is all necessary work that we need to do before we write line one of automation. All of those are critical, sapient tasks, and critical thinking, sapient testers are valuable in this process, and if we leverage the opportunities, we can make ourselves indispensable.

---

11:30 am - 12:30 pm
PERFORMANCE TESTING IN AGILE CONTEXTS
Eric Proegler

the other title for this talk is "Early Performance Testing", and a lot of the ideas Eric is advocating is to look for ways to front load performance testing rather than wait until the end and then worry about optimization and rework. this makes a lot of sense when we consider that getting performance numbers early in development means we can get real numbers and real interactions. It's a great theory, but of course the challenge is in "making it realistic". Development environments are by their very nature not as complete or robust as a production environment. In most cases, the closest we can come to is an artificial simulation and a controlled experiment. It's not a real life representation, but it can still inform us and give us ideas as to what we can and should be doing.

One of the valuable systems we use in our testing is a duplicate of our production environment. In our case, when I say production, what I really mean is a duplicate of our staging server. Staging *is* production for my engineering team, as it is the environment that we do our work on, and anything and everything that matters to us in our day to day efforts resides on staging. It utilizes a lot of the things that our actual production environment uses (database replication, HSA, master slave dispatching, etc.) but it's not actually production, nor does it have the same level of capacity, customers and, most important, customer data.

Having this staging server as a production basis, we can replicate that machine and, with the users, data and parameters as set, we can experiment against it. Will it tell us performance characteristics for our main production server? No, but it will tell us how our performance improves or degrades around our own customer environment. In this case, we can still learn a lot. By developing performance tests against this duplicate staging server, we can get snapshots and indications of problem areas we might face in our day to day exercising of our system. What we learn there can help inform changes our production environment may need.

Production environments have much higher needs, and replicating performance, scrubbing data, setting up a matching environment and using that to run regular tests might be cost prohibitive, so the ability to work in the small and get a representative look can act as an acceptable stand in. If our production server is meant to run on 8 parallel servers and handle 1000 consecutive users, we may not be able to replicate that, but creating an environment with one server and determining if we can run 125 concurrent connections and observe the associated transaction can provide a representative value. We may not learn what the top end can be, but we can certainly determine if problems will occur below the single server peak. If we discover issues here, it's a good bet production will likewise suffer at its relative percentage of interactions.

How about Perfomance Testing in CI? Can it be done? It's possible, but there are also challenges. In my own environment, were we to do performance tests in our CI arrangement, what we are really doing is testing the parallel virtualized servers. It's not a terrible metric, but I'd be leery of assigning authoritative numbers since the actual performance of the virtualized devices cannot be guaranteed. In this case, we can use trending to see if we either get wild swings, or if we get consistent numbers with occasional jumps and bounces.

Also, we can do performance tests that don't require hard numbers at all. We can use a stop watch, watch the screens render, and use our gut intuitions as to whether or not the system is "zippy" or "sluggish". They are not quantitative values, but they have a value, and we should leverage our own senses to encourage further explorations.

The key takeaway is that there is a lot we can do and there's a lot of options we have so that we can make changes and calibrate our interactions and areas we are interested in. We may not be able to be as extensive as we might be with a fully finished and prepped performance clone, but there's plenty we can do to inform our programmers as to the way the system is behaving under pressure.

---

1:15 pm - 1:45 pm
KEYNOTE: THE MEASURES OF QUALITY
Brad Johnson

One of the biggest challenges we all face in the world of testing is that quality is wholly subjective. there are things that some people care about passionately that are far less relevant to others. The qualitative aspects are not numerable, regardless of how hard we want to try to do so. Having said that, there are some areas that counts, values, and numbers are relevant. to borrow from my talk yesterday, I can determine if an element exists or if it doesn't. I can determine if the load time of a page. "Fast" or "slow" are entirely subjective, but if I can determine that it takes 54 milliseconds to load an element on a page as an average over 50 loads, that does give me a number. The next question of course is to think "is that good enough?" It may be if it's a small page with only a few elements. If there are several elements on the page that take the same amount of time to serially load, that may prove to be "fast" or "slow".

Metrics are a big deal when it comes to financials. We care about numbers when we want to know how much stuff costs, how much we are earning, and to borrow an oft used phrase "at the end of the day, does the Excel line up?" If it doesn't, regardless of how good our product is, it won't be around long. Much as e want to believe that metrics aren't relevant. Sadly they are, in the correct context.

Testing is a cost. Make no mistake about it. e don't make money for the company. We can hedge against losing money, but as testers, unless we are selling testing services, testing is a cost center, it's not a revenue center. To the financial people, any change in our activities and approaches is often looked at in the costs those changes will occur. their metric is "how much will this cost us?" Our answer needs to be able to articulate "this cost will be leveraged by securing and preserving this current and future income". Glamorous? Not really, but its essential.

What metrics do we as testers actually care about, or should we care about? In my world view, I use the number of bugs found vs number of bugs fixed. That ratio tells me a lot. This is, yet again,  a drum I hammer regularly, and this should surprise no one when I say I personally value the tester whose ratio of bugs reported to bugs fixed is closest to 1:1. Why? It means to me that testers are not just reporting issues, but that they are advocating for their being fixed. Another metric often asked about is the number of test cases run. Tome, it's a dumb metric, but there's an expectation outside of testing that that is informative. We may know better, but how do we change the perspective? In my view, the better discussion is not "how many test cases did you run" but "what tests did you develop and execute relative to our highest risk factors?" Again, in my world view, I'd love to see a ratio of business risks to test charters completed and reported to be as close to 1:1 as possible.

In the world of metrics, everything tends to get boiled down to Daft Punk's "Harder Better Faster Stronger". I use that lyrical quote not just to stick an ear-worm in your head (though if I have you're welcome or I'm sorry, take your pick), but it's really what metrics mean to convey. Are we faster at our delivery? Are we covering more areas? Do we finish our testing faster? Does our deployment speed factor out to greater revenue? One we either answer Yes or No, the next step is "how much or how little, how frequent or infrequent. What's the number?"

Ultimately, when you get to the C level execs, qualitative factors are tied to quantitative numbers, and most of the time, the numbers have to point to a positive and/or increasing revenue. That's what keeps companies alive. Not enough money, no future, it's that simple.

Brad suggests that, if we need to quantify our efforts, these are the ten areas that will be the most impactful.

It's a pretty good list. I'd add my advocacy and risk ratios, too, but the key to all of this is these numbers don't matter if we don't know them, and they don't matter if we don't share them.

---

2:00 pm - 3:00 pm
TESTING IS YOUR BRAND. SELL IT!
Kate Falanga


One of the oft heard phrases that is heard among software testers and about software testing is that we are misunderstood. Kate Falanga is in some ways a Don Draper of the testing world. She works with Huge, Huge is like Mad men, just more computers, though perhaps equal amounts of alcohol ;). Seriously, though, Kate approaches software testing as though it were a brand, because it is, and she's alarmed at the way the brand is perceived. The fact is, every one of us is a brand unto ourselves, and what we do or do not do affects how that brand is perceived.

Testers are often not very savvy about marketing themselves. I have come to understand this a great deal lately. The truth is, many people interpret my high levels of enthusiasm, my booming voice, and my aggressive passion and drive to be good marketing and salesmanship. It's not. It can be contagious, it can be effective, but that doesn't translate to good marketing. Once that shine wears off, if I can't effectively carry objectives and expectations to completion, or to encourage continued confidence, then my attributes matter very little, and can actually become liabilities.

I used to be a firebrand about software testing and discussing all of the aspects about software testing that were important... to me. Is this bad? Not in and of itself, but it is a problem if I cannot likewise connect this to aspects that matter to the broader organization. Sometimes my passion and enthusiasm can set an unanticipated expectation in the minds of my customers, and when I cannot live up to that level of expectation, there's a let down, and then it's a greater challenge to instill confidence going forward. Enthusiasm is good, but the expectation has to be managed, and it needs to align with the reality that I can deliver.

Another thing that testers often do is they emphasize that they find problems and that they break things. I do agree with the finding problems, but I don't talk about breaking things very much. Testers generally speaking don't break things, we find where they are broken. Regardless of how taht's termed, it is perceived as a negative. It's an important negative, but it's still seen as something that is not pleasant news. Let's face it nobody wants to hear their product is broken. Instead, I prefer, and it sounds like Kate does to, that emphasizing more positive portrayals of what we do is important. Rather than say "i find problems", instead, I emphasize that "I provide information about the state of the project, so that decision makers can make informed choices to move forward". Same objective, but totally different flavor and perception. Yes I can vouch for the fact that the latter approach works :).

The key takeaway is that each of us, and by extension our entire team, sells to others an experience, a lifestyle and a brand. How we are perceived is both individual and collectively, and sometimes, one member of the team can impact the entire brand, for good or for ill. Start with yourself, then expand. Be the agent of change you really want to be. Ball's in your court!

---

3:15 pm - 4:15 pm
BUILDING LOAD PROFILES FROM OBJECTIVE DATA
James Pulley

Wow, last talk of the day! It's fun to be in a session with James, because I've been listening to him via PerfBytes for the past two years, so much of this feels familiar, but more immediate. While I am not a performance tester directly, I have started making strides to start getting into this world because I believe it to be valuable in my quest to be a "specializing generalist" or a "generalizing specialist" (service mark Alan pAge and Brent Jensen ;) ).

Eric Proegler in his talk earlier talked about the ability to push Performance testing earlier in the development and testing process. To continue with that idea, I was curious to get some ideas as to how to build a profile to actually run performance and load testing. can we send boatloads of request to our servers and simulate load? Sure, but will that actually be representative of anything meaningful? In this case, no. What we really want to create is a representative profile of traffic and interactions that actually approaches the real use of our site. To do that, we need to think what will actually represent our users interactions with our site.

That means that workflows should be captured, but how can we do that? One of the ways that we can do this is analysis of previous transactions with our logs, and recreating steps and procedures. Another way is to look at access or error logs to see what people want to find but can't, or see if there are requests that look to not make any sense (i.e. potential attacks on the system). The database admin, the web admin and the CDN Administrator are all good people to cultivate a relationship with to be able to discuss the needs and encourage them to become allies.

Ultimately, the goal of all of this is to be able to steer clear of the "ugly baby syndrome" and look to cast blame or avoid blame, and to do that, we really need to make it possible to be as objective as possible. with a realistic load of transactions that are representative, there's less of a chance for people to say "that test is not relevant" or "that's not a real world representation of our product".

Logs are valuable to help gauge what actually matters and what is junk, but those logs have to be filtered. there are many tools available to help make that happen, some commercial, some open source, but the goal is the same, look for payload that is relevant and real. James encourages looking gat individual requests to see who generated requests, who referred the request, what request was made, and what user agent made the request (web vs mobile, etc.). What is interesting is that we can see patterns that will show us what paths users use to get to our system and what they traverse in our site to get to that information. Looking at these traversals, we can visualize pages and page relationships, and perhaps identify where the "heat" is in our system.

---

Wow, that was an intense and very fast two full days. My thanks to everyone at STP for putting on what has been an informative and fun conference. My gratitude to all of the speakers who let me invade their sessions, type way too loudly at times (I hope I've been better today) and inject my opinions here and there.

As we discussed in Kate's session, change comes in threes. The first step is with us, and if you are here and reading this, you are looking to be the change for yourself, as I am looking to be the change in myself.

The next step is to take back what we have learned to our teams, openly if possible, by stealth if necessary, but change your organization from the ground up.

Finally, if this conference is helpful and you have done things that have proven to be effective, had success in some area, or you are in an area that you feel is under representative, take the third step and engage at the community level. Conferences need fresh voices, and the fact is experience reports of real world application and observation have immense value, and they are straightforward  talks to deliver. Consider putting your hat in the ring to speak at a future session of STP-CON, or another conference near you.

Testing needs all of us, and it will only be as good as the contributors that help build it. I look forward to the next time we can get together in this way, and see what we can build together :).

In My Own Time: A Live Blog from #STPCON, Spring 2015

To borrow a favorite line from my daughters... 

"What does snow turn into when it melts? It becomes spring!" 

It's a line from one of their all time favorite Anime series (Fruits Basket), and it's a cute ring of optimism that points to warmer days, longer daylight and at least for me, the start of the presenting and writing season. 

Over the past few years, I've enjoyed a three times per year experience of a conference in spring, CAST in the summer, and a conference a little farther afield in the fall. I've been invited to both Sweden and Ireland in the past year, and I'm keeping my fingers crossed that I might have an opportunity to present in Germany in the fall of 2015. Right now, though, that's up in the air, but I can hope and dream.

Today, though, brings me to slightly cloudy but very pleasant San Diego for the regular sessions of STPCON Spring 2015. Monday and Tuesday were special sessions and tutorials, and due to work commitments, I couldn't get here until last night. A group of us were gathered around the fire pit outside in the later evening, though, talking about our respective realities, and that familiar "conferring before conferring" experience that we all know and love. 

A great surprise this morning was to see my friend Mike Ngo, a tester who worked with me a decade ago at Konami Digital Entertainnment. Many of us have stayed in touch with each other over the years; we jokingly refer to ourselves as being part of the "ExKonamicated". He told me he came with a number of co-workers, and that several of them would be attending my talk this morning (oh gee, no pressure there ;) ).

Mike Lyles stated of the program today welcoming testers and attendees coming far and wide (apparently Sweden sent the most people, with 12 attendees), and many companies sending contingents (Mike's team from Yammer being a good example). The food was good this morning, the company has been great, and as always, it's such great fun to see people that I really only get to see online or at these events. By the way, a quick plug for Fall 2015 STPCON. It will be on a cruise ship ("Hey Matt, it's happened. They're having a testing conference... on a boat ;)... HA HA, April Fools!!!" Ah well, therein lies the danger of presenting on April 1st ;) ).

---

Bob Galen starts off the day with a keynote address defining and describing the three pillars of Agile Quality and Testing. As many of us can attest (and did so, loudly), there are challenges with the way Agile Development and Agile Testing take place in the real world. Interestingly enough, one of the summing points is that we have a lot of emphasis on "testing", but that we have lost the focus on actual "quality". In some ways, the fetish with automated testing and test driven development (which admittedly, isn't really testing in the first place, but the wording and the language tends to make us think it does) has made for "lots of testing, but really no significant differences in the perceived quality of the applications as they make their way out in the world". We have Selenium and Robot Framework and Gherkin, oh my, but could these same teams write anything that resembled a genuine Acceptance Test that really informed the quality of the product? So many of these tests provide coverage, but what are hey actually covering?

The biggest issue with all of our testing is that we are missing the real "value perspective". Where are our conversations leading? What does our customer want? What are we actually addressing with our changes? Story writing is connected to the automation, and that's not essentially a bad thing, but it needs to be a means to an end, not the end in and of itself.

The three pillars that Bob considers to be the core of successfully balanced Agile practices are Development & Test Automation, Software Testing, and Cross Functional Team Practices. Agile teams do well on the first pillar, are lacking in the second, and in many ways are really missing the third. The fetish of minimizing needless and wasteful documentation has caused some organizations to jettison documentation entirely. I frequently have to ask "how has that worked for you?". I'm all for having conversations rather than voluminous documentation, but the problem arises when we remove the documentation and still don't have the conversations. Every organization deals with this in some way when they make an Agile transition. 

Beyond the three pillars, there's also the sense that there is a foundation that all Agile teams need to focus on. Is there a whole team ownership of quality? Are we building the right things? Are we actually building it right? If we are using metrics, are they actually useful, and do they make sense? How are we steering the quality initiatives? Are we all following the same set of headlights? Are we actually learning from our mistakes, or are we still repeating the same problems over and over again, but doing it much faster than ever before?

Agile is a great theory (used in the scientific manner and not the colloquial, I genuinely mean that), and it offers a lot of promise for those that embrace it and put it into practice (and practice is the key word; it's not a RonCo product, you don't just "set it and forget it"). It's not all automation, it's not all exploratory testing, and it's not all tools. It's a rhythm and a balance, and improvising is critical. Embrace it, learn from it, use what works, discard what doesn't. Rinse and repeat :).

---

The last time I spoke at STP-CON, I was the closing speaker for the tracks. This time around, I was one of the first out the gate. If I have to choose, I think I prefer the latter, though I'll speak whenever and wherever asked ;). I've spoken about Accessibility in a number of places, and on this blog. Albert Gareev and I have made it a point of focus for this year, and we have collaborated on talks that we are both delivering. When you see me talking about Accessibility, and when you see Albert talk about it, we are usually covering the same ground but from different perspectives. I delivered a dress rehearsal of this talk at the BAST meetup in January in San Francisco, and I received valuable feedback that made its way into this version of the talk.

The first change that I wanted to make sure to discuss right off the bat is that we discuss Accessibility from a flawed premise. We associate it as a negative. It's something we have to do, because a government contract mandates it or we are being threatened with a lawsuit. Needless to say, when you approach features and changes from that kind of negative pressure, its easy to be less than enthusiastic, and to take care of it after everything else. Through discussions I had with a number of people at that dress rehearsal talk, we realized that a golden opportunity to change the narrative was already happening. An attendee held up their smart phone and said "this, right here, will drive more changes to accessible design than any number of government regulations", and they were absolutely right! We see it happening today with micro services being developed to intake data from wearable wearables like the Apple Watch, the FitBit, or other peripheral devices, where general text and screens just aren't the way the product needs to interact.

While I was developing this talk, and sent my slides to be included in the program, a brilliant tweet was made by @willkei on Twitter that encapsulated this interesting turning point:

Yeah, that!

The same steps and processes being used to make content accessible on our smart phones and our smart watches and the Internet of Things are really no different than designing for accessibility from the outset. The form factor necessitates the need for simplicity and direct communication, in a way that a regular interface cannot and will not do. What's great about these developments is the fact that new methods of interaction are having to be developed and implemented. By this same process, we can use this opportunity to incorporate those design ideas for everyone.

A heuristic that Albert and I (well, mostly Albert) have developed that can help testers think about testing for accessibility is called HUMBLE .

HUMBLE

Humanize: be empathetic, understand the emotional components.  

Unlearn: step away from your default [device-specific] habits. Be able to switch into different habit modes.

Model: use personas that help you see, hear and feel the issues. Consider behaviors, pace, mental state and system state.

Build: knowledge, testing heuristics, core testing skills, testing infrastructure, credibility.

Learn: what are the barriers? How do users Perceive, Understand and Operate?

Experiment: put yourself into literal situations. Collaborate with designers and programmers, provide feedback

In short, Accessible design isn't something special, or an add on for "those other people" out there. It's designing in a way that allows the most people the ability to interact with a product effectively, efficiently and in a way that, we should hope, includes rather than excludes. Designing for this up front is much better, and easier to create and test, then having to retrofit products to be accessible. Let's stop looking at the negative associations of accessibility, because we have to do it, and instead focus on making a web that is usable and welcoming for all.

---

Maaret Pyhäjärvi covered a topic that intrigued me greatly. While there are many talks about test automation and continuous delivery, Maaret is approaching the idea of "Continuous Delivery WITHOUT Automation". Wait, what?

Some background for Maaret's product would probably be helpful. She's working on a web based .NET product on a team that has 1 tester for each developer, and a scrum like monthly release for two years. As might be guessed, releases were often late and not tested well enough. This created a vicious cycle of failing with estimates and damaging team morale which, you guessed, caused them more failing with estimates. 

What if, instead of trying to keep working on these failed estimates and artificial timelines, we were to move to continuous deployment instead? What was interesting to me was the way that Maaret described this. They don't want to go to continuous delivery, but instead, continuous deployment. the build, unit test, functional test and integration test components were all automated, but the last step, the actual deployment, is manually handled. Instead of waiting for a bunch of features to be ready and push to release, each feature would be individually released to production as it is tested, deemed worthy and ready to be released. 

This model made me smile because in our environment, I realized that we did this, at least partially. Our staging environment is our internal production server. It's what we as a company actively use to make sure that we are delivering each finished feature (or sometimes plural features) and deploying the application on a regular basis. 

Maaret discussed the value of setting up a Kanban board to track features as a one piece flow. We likewise do this at Socialtext. Each story makes its way through the process, and as we work on each of the stories, as we find issues, we address the issues until all of the acceptance criteria passes. From there, we merge to staging, create a build, and then we physically update the staging server, sometimes daily, occasionally multiple times a day. This helps us to evaluate the issues that we discover in a broader environment and in real time vs. inside of our more sterile and generalized test environments. It seems that Maaret and her team do something very similar.

What I like about Maaret's model, and by extension, ours, is the fact that there remains a human touch to the deployments so that we can interact with and see how these changes are affecting our environment in real time. We remain aware of the deployment issues if they develop, and additionally, we share that role each week. We have a pumpking that handles the deployment of releases each week, and everyone on the engineering team takes their turn as the pumpking. I take my turn as well, so I am familiar with all of the steps to deploy the application, as well as places that things could go wrong. Having this knowledge allows me to approach not just my testing but the actual deployment process from an exploratory mindset. It allows me to address potential issues and see where we might have problems with services we depend on, and how we might have issues if we have to update elements that interact with those services independently.

---

Paul Grizzaffi  has focused on the creating and deploying automated test strategies and frameworks and emphasizes in his talk about changing the conversation that we are having about test automation and how we approach it. in "Not Your Parents’ Automation – Practical Application of Non-Traditional Test Automation", we have to face the fact that traditional automation does matter. we use it for quantitative items and both high and low level checks. However, there's other places that automation can be applied. How can we think differently when it comes to software test automation?

Often, we think of traditional automation as a way to take the human out of the equation entirely. That's not necessarily a bad thing. There's a lot of tedious actions that we do that we really wish we didn't have to do. when we build a test environment, three are a handful of commands I now run, and those commands I put into a script so I can do it all in one step. That's a good use of traditional automation, and it's genuinely valuable. We often say we want to take out the drudgery so the human can focus on the interesting. Let's extend this idea, can we use automation to help us with the interesting as well?

Paul asks  a fundamental question; “how can automation help me do my job?” When we think of it that way, we open new vistas and consider different angles where our automation can help us look for egregious errors, or look at running a script on several machines in parallel. How about if we were to take old machines that we could run headless and create a variety of test environments, and run the same scripts on a matrix of machines. Think that might prove to be valuable? Paul said that yes, it certainly was for their organization.

Paul uses a term I like and have used as a substitute for automation. I say Computer Assisted Testing, Paul calls it Automation Assist, but overall it's the same thing. we're not looking for atomic pass fail check validations. Instead we are looking for ways to observe and use our gut instincts to help us consider if we are actually seeing what we hope to see or determine that we are not seeing what we want to. 

At Socialtext, I have been working on a script grouping with this in mind. We call it Slideshow, and it's a very visible automation tool. For those who recall my talk about balancing Automation and Exploratory Testing, I used the metaphor of a taxi cab in place of a railroad train. The value of a taxi cab is that it can go many unique places,and then let you stop and look around. Paul's philosophy fits very well with that approach and use. Granted, it's fairly labor intensive to make and maintain these tests because we want to keep looking for unique places to go and find new pages. 

Some key ideas that Paul asks us to consider, and I feel makes a lot of sense, is that automation is important, but it gets way too much focus and attention for what it actually provides. If we don't want to scale back our traditional automation efforts, then we should look for ways that we can use it in unique and different ways. One of my favorite statements is to say "forget the tool for now, and start with defining the problem". By doing that, we can then determine what tools actually make sense, or if a tool is actually needed (there's a lot you can do with bash scripts, loops and curl commands... just sayin' ;) ). Also, any tool can be either an exceptional benefit or a huge stumbling block. It depends entirely on the person using the tool, and their expertise or lack thereof. If there is a greater level of knowledge, then more creative uses are likely to result, but don't sell short your non toolsmith testers. They can likely still help you find interesting and creative uses for your automation assistance.

---

I think Smita Mishra deserves an award for overachieving. Not content to deliver an all day workshop, she's also throwing down with a track talk as well. For those who don't know Smita, she is the CEO and Chief Test Consultant with QAzone Infosystems. She's also an active advocate, writer and engaged member of our community, as well as being a rock solid software tester. In her track talk, she discussed "Exploratory Testing: Learning the Security Way".

Smita asks us to consider what it would take to perform a reasonably good security test. Can we manually test security to a reasonable level? Can exploratory testing give us an advantage?

First off, Exploratory Testing needs to be explained as to what it is not. It's not random, it's not truly a ad hoc, at least not if it is being done correctly. A favorite way I like to look at Exploratory Testing is to consider interviewing someone, asking them questions, and based on their answers, asking follow up questions that might lead into interesting and informative areas. Like a personal interview, if we ask safe boring questions with predictable answers, we don't learn very much. However, when we ask a question that makes someone wince, or if we see they are guarded with their answers, or even evasive, we know we have hit on something juicy, and our common desire is to keep probing and find out more.  Exploratory testing is much the same, and security testing is a wonderful way to apply these ideas. 

One of the most interesting ways to attack a system is to determine the threat model and the prevalence of the attack. SQL injections are ways that we can either confirm security of forms fields or se if we can compromise the fields and gain access. 

What is interesting about security testing as a metaphor for exploratory testing is the fact that attacks are rarely tried one off and done. If an issue doesn't manifest with one attack, a hacker will likely try several more. As a tester we need to think and try to see if we can approach the system in a similar manner. It's back to the interviewing metaphor, with a little papparazzi determination for good measure. 

Ask yourself what aspects of the application would make it vulnerable. What services reside in one place, and what services reside in another? Does your site use iFrames? Are you querying different systems? Are you exposing the query payload? Again, each answer will help inform the next question that we want to ask. Each venue we open will allow us to find a potential embarrassment of riches. 

Smita encourages use of the Heuristic Test Strategy Model, which breaks the product into multiple sections (Quality Criteria, Project Environment, Product Elements and Test Techniques, as well as the Perceived Quality that the product should exhibit). Each section has a further breakdown of criteria to examine, and each breakdown thread allows us to ask a question, or group of questions. This model can be used for something as small as a button element or as large as an entire application. Charters can be developed around each of the questions we develop, and our observations can encourage us to ask further clarifying questions. It's like a puzzle when we start putting the pieces together. Each successful link-up informs the next piece to fall in place,and each test can help us determine which further tests to perform, or interlocking areas to examine.

---

Our closing keynote talk today is being delivered courtesy of Mark Tomlinson, titled "It’s Non-Functional, not Dysfunctional". There's functional testing and the general direct interaction we are all used to considering, and then there's all that "other" stuff, the "ilities" and the para-fucnctional items. Performance, Load, Security, Usability, Accessibility, and other items all play into the testing puzzle, and people need to have these skills. However, we don't have the skills readily available in enough places to actually give these areas the justice it deserves. 

Programmers are good at focusing on the what and the where, and perhaps can offer a fighting chance at the why, but the when and the how sometimes fall by the wayside.  Have we considered what happens when we scale up the app? Does the database allow enough records? Can we create consitions that bring the system to its knees (and yes, I know the answer is "yes" ;) ). 

there are a variety of dysfunctions we need to address, and Mark broke down a few here:

Dysfunction 1: Unless the code is stable, I can't test performance.

What determines stable? Who determines it? How long will it take to actually make it stable? Can you answer any of those? Functional testers may not be able to, but performance testers may be able to help make unstable areas more stable (mocks, stubs, virtual services, etc.). We can't do a big run of everything, but we can allocate some effort for some areas.

Dysfunction 2: As a product manager, I've allocated 2 weeks at the end of the project for load testing.

Why two weeks? Is that anywhere near enough? Is it arbitrary? Do we have a plan to actually do this? What happens if we discover problems? Do we have any pivoting room?

Dysfunction 3: The functional testers don't know how to use the tools for performance or security.

Considering how easy it is to download stuff like jMeter and Metasploit or Kali Linux, there's plenty of opportunities to do "ilitiy" testing for very close to free if not outright free. If you happen to get some traction, share what you know with others, and suddenly, surprise, you have a tool stack that others can get involved with.

Dysfunction 4: We don't have the same security configuration in TEST that we do in PROD.

Just because they don't match exsctly doesn't mean that the vulnerabilities are not there. There's a number of ways to look at penetration testing and utilize the approaches these tools use. What if the bizarre strings we use for security testing were actually used for our automated tests? Insane? Hardly. It may not be a 1:1 comparison, but you can likely use it.

Dysfunction 5: Our test environment is shared with other environments and hosted on over provisioned virtual server hosts.

OK, yeah, that's real, and that's going to give us false positives (even latency in the cloud can muck some of this up). this is also a chance to communicate times and slices to access the machines without competing for resources. 

Dysfunction 6: We don ot have defined SLA's or non-functional requirements specifications.

Fair enough, but do we even have scoping for one user? Ten users? One hundred users? What's the goal we are after? Can we quantify something and compare with expectations? Can we use heuristics that make sense to determine if we are even in the ball park?

Question A:

Question B:

Would we approach these questions the exact same way? Ha! Trick question! Of course we wouldn't, but at least by structuring the questions this way (Question B) we can make decisions that at least allow us to make sure we are asking the right questions in the first place. More to the point, we can reach out to people that have a clue as to what we should be doing, but we can't do that if we don't know what to ask.

---

This concludes the Wednesday session of STP-CON. Join me tomorrow for "Delivering the Goods: A Live Blog from #STPCON".