Data: What, Why, How??: an #STPCon Live Blog Entry

All good things must come to an end and due to a need to pick up family members from the airport today, this going to have to be my last session for this conference. I'm not entirely sure why this is but Smita seems to be in the closing sessions at STP events. Thus Smita Mishra will be the last speaker I'll be live blogging for this conference. My thanks for everyone involved in putting on a great event and for inviting me to participate. With that, let's close this out.

Data is both ubiquitous and mysterious. There are a lot of details we can look at and keep track of. There is literally a sea of data surrounding our every action and expectation. What is important is not so much the aggregation of data but the synthesis of that data, making sense of the collection, separating the signal from the noise.

Data Scientists and Software testers share a lot of traits. In many cases, they are interchangeable terms. Data and the analysis of data is a critical domain inside the classic scientific method. Without data, we don't have sufficient information to make an informed decision.

When we talk about "Big Data" what we are really looking at are the opportunities to parse and distill down all of the data that surrounds us and utilize it for special purposes. Data can have four influences:

• Processes
• Tools
• Knowledge
• People

Data flows through a number of levels, from chaos to order, from Uncertainty to certainty. Smita uses the following order; Uncertainty, Awakening, Enlightenment, Wisdom, Certainty. We can also look at the levels of quality for data: Embryonic, Infancy, Adolescence, Young Adult, Maturity. To give these levels human attributes, we can use the following; Clueless, Emerging, Frenzy Stabilizing Controlled. In short, we move from immaturity to maturity, not knowing to know for certain, etc.

Dat can likewise be associated with its source; Availability, Usability, Reliability, Relevance and the ability to present the data. Additionally, it needs to have an urgency to be seen and to be synthesized. Just having a volume of data doesn't mean that it's "Big Data". The total collection of tweets is a mass of information. It's a lot of information to be sure but it's just that, we can consider it "matter unorganized". The process of going from unorganized matter to organized matter is the effective implementation of Big Data tools.

Dat can sound daunting but it doesn't need to be scary. It all comes down to the way that we think about it and the way that we use it.

Testing Monogatari: a #STPCon Live Blog Entry

Ah yes, the after lunch stretch. I will confess I'm a tad bit drowsy but what better way to cure that than to have a rapid-fire set of five-minute test talks.

We've had several speakers play along with a variety of topics:

Brian Kitchener getting hired into a QTP shop with a background in Selenium. His idea was talking about how to make organizational changes when they are a big ship and you are just one person. The answer is lots of little changes to help prove that making small scale changes and proving their benefits can be scaled to make larger changes.

Brian Saylor talked about the idea that "CAN'T" is a four letter word. To Brian, can't is the most offensive word.

"I can't do that" really means "I don't want to do that".
"It can't be done" often means "I'm too lazy to do this"
"You can't do that" often means "I can't do that, so therefore you shouldn't be able to"

Thus Brian asks us to be sensitive and to strike the word "can't" from our vocabulary and see what it is we are really saying.

Tricia Swift talked about changes she has seen over the last twenty years. The major changes she noticed was that there was about ten percent of her MIS class who were women. Compared to her CS friends, she was a lot more represented. She is happy to see that that has changed. ISO compliance and Waterfall was everything twenty years ago (and oh do I remember *that*). Thankfully, much of that has changed, or at least the approach has changed. Most important, she is seeing women in development where twenty years ago there were very few.

Raj Subramanian wants to make it clear that "It Is OK If You Mess Up". The issue with a culture where messing up is punished means that nothing creative or innovative will be tried. Cultures that penalize people for making mistal=kes ensure that they will remain stagnant. Raj basically shared an example where a test had an effect on production (an airline app) and the learning experience was that his rogue test exposed a series of policy problems with their production server. Raj still has that job and the product was improved, all because a mistake was acknowledged and accepted.

Anan Parakasan shared his experiences with the conference and the development and focus on "trust". It's a word that has a lot of different meanings and isn't exactly easy to nail down. Anan shared ways that he felt that trust could be better fostered and developed for our teams. Additionally, developing Emotional Intelligence will help considerably by making a team more effective. Think less adversarial and more collaboratively.

Final talk is Heath (?) and he talked about "feature flag insanity" and the fact that his organization has gone a bit trigger happy with their feature flags. Having them means running with them on, of and not there. However, his point was that the flag had contexts that were understood by some people but not others. Needless to say, the feature flag had an option to fail every credit card transaction. Not being able to find it and have it rolled out to production meant that every credit card transaction in production was failed and that cost a whole lot of money. In short, know your system, document it and make sure everyone knows what the feature flags do.

Came down to Raj and Brian Kitchener and they both have $50 Amazon gift cards. And with that, I'm much more awake ;).

Making the Move to Continuous Testing: a #STPCon Live Blog Entry

Sorry for the gap in the 10:00 hour but I took advantage of the opportunity to talk with a vendor that we actually work with during that hour to discuss an issue we've been dealing with. Kudo to them for taking the time, letting me vent and demonstrate and figuring out next steps.

With that, I'm definitely interested in seeing what is happening with Continuous Testing. I first started reading about this topic almost ten years ago and it is interesting that it is still a hot button topic.

We do a lot with Continuous Integration (CI) where I am and have made strides with Continuous Delivery (CD). Now I'm curious as to the possibility of implementing Continuous Testing (CT).

Alissa Lydon works with Sauce Labs, and she is starting with getting a better understanding of what CT actually means. It doesn't mean that automated tests are running constantly (though automation is a part of the discussion). It does mean testing at every stage of development.

The first step to getting CT to have a fighting chance is that everyone has to be committed to quality, which means every level of the development process needs to think about testing and staying ahead of the curve. Think of having programmers and testers sitting together, pair navigating and pair programming/testing to help get testing principles and approaches into the process of writing software before line one of code even gets written. The key is that everyone needs to be on the same page and as early as possible.

Step two is testing at every stage of the development cycle. No code should pass through any stage of development without some level of testing being performed. This can range everywhere from requirement provocations to unit tests to integration tests to end-to-end tests, whatever makes sense at the given point in time.

Let's segue here and mention that continuous testing does not necessarily mean automated testing. Automation can make sense at a variety of points but you can have a mix of automated and live/manual tests and still be considered CT. There is also a balance of the levels of testing that will be done at any given time.

Next is being able to leverage the skills of the team so that automated testing can advance and get put in place. While Automation is not the sole criteria for CT, it is definitely important to be able to make CT work.  It will take time and attention and it will certainly take some development chops. Automation needs to be treated every bit as important as the production code and I'll fight anyone who says otherwise ;).

Additionally, a breadth of devices and platforms are critical to getting a realistic view of how your environment will look on a broad cross-section of resolutions and sizes, as well as user agents.

The ability to scale the systems being tested is important. Yes, we may start with just one machine, but ideally, we want to be able to simulate our production environment, even if we go with a smaller number of machines and try to extrapolate what a larger number would provide.

The final step is implementing and actualy using/reviewing analytics. In short, know what your product is doing and what is being used to focus efforts.

Testing all the World's Apps with AI: an #STPCon Live Blog Entry

We are here on the last day of STP-Con Spring 2019. Today is going to be nice for me in that I won't have any responsibilities short of listening and typing these blog posts. It feels so good to be "done" when it comes to speaking. I enjoy it, don't get me wrong, but there's a level of focus and energy that takes up all of your focus until you're finished. I'm happy to be on the other side for the rest of the conference.


This morning we are listening to Jason Arbon talking about testing with AI. Part of the puzzle to use AI is to teach machines the following concepts:

See
Intent
Judge
Reuse
Scale
Feedback

These are concepts our machines have to come to grips with to be able to leverage AI and have us gain the benefits of using it. Our machines need to be able to see what they are working with, they need to be able to determine the intent of tests, to judge their fitness and purpose (machines making judgment calls? This might make a number of my long-held assertions obsolete ;) ). Machines need to be able to reuse the code and the learnings that they have been able to apply. The systems need to be able to scale up to run a lot of tests to gather a representative set of data. Finally, the scripts and apps need to be able to respond to the feedback loop.

Jason is showing examples of "Automated Exploratory Testing". Wait, what? well, yes, there are some things we can do that would certainly correspond with what we would call classic exploratory testing. At least we can look at a variety of examples of what we would normally do. Sure, a human could be super random and do completely whacked out things, but most of us will use a handful of paths and approaches, even if we are in exploratory mode. Thus, it is possible to look at a lot of "what if" scenarios. It requires a lot of programming and advanced work, but at some point, we can collect a lot of data to do certain things. There's a convergence force at play so yeah, I guess bots can do exploratory testing too :).

Using Agile to Adapt to Changing Goals: an #STPCon Live Blog Entry

Using Agile to Adapt to Changing Goals

There's an old joke that goes around that says, basically, "If you don't like the way things are today, come back tomorrow". Meaning the only true constant is change and change happens a lot. We all have choices to be a driver of change or to have change be driven onto us. To this idea, any goals that your team has made are just as likely to change.


Sue Jagodzinski described her own situation of change within her company and the chaos that resulted from it. Part of her story described the challenges of managing two teams that were focused on different things. Both teams needed to use and support the adoption of a test automation tool and to be able to enhance and support that tool. One team focused on build, test, and automation. The other team focused on training, support, and documentation. While the tool they were working on was the same, they had very different missions and purposes and that would show up in the backlogs they built up and the priorities they placed on their respective backlogs.

Here's a bit of nerdy for you. The training and support team decided to call themselves Team1. In retaliation, the other team called themselves Team0 because they understood how arrays were indexed (and yes, this was a real dynamic between these teams. I thought she was kidding, she assured me she was not.

To this end, Sue determined that there were some key danger signs happening here. By segregating responsibility between two teams, there was an unhealthy competition that developed. trust issues developed along the way. when there issues there was plenty of finger pointing to go along with it. Most visibly was the fact that there were conflicts where teams would determine what to work on was guided by whatever the other team was not working on.

By moving to a shared backlog, the teams came into immediate conflict and had to negotiate how to change that dynamic. Some of the areas that Sue addressed was:

- how can we determine the skills that were needed on each team, and institute necessary moves if needed?
- determine the soft skill levels of the team members, who were the leaders? Who could become a new leader if needed?
- who would be best served to go elsewhere?
- how could we restructure the teams to be less contentious?

The last one was easy-ish to solve by changing names. the two teams for purposes of this presentation were "Awesomation" and "Thorium". Both teams agreed to go to a single backlog. Teams were set up so that both teams had technical expertise in the test framework. More to the point, an emphasis was made to reward and encourage those that would share their knowledge and skill sets. By doing this Sue was able to get close to equalizing the team. By sharing a backlog, the grooming sessions were done together, with their expected challenges. Sue took over as the product owner and she had to learn what that entailed. She described the process as being "harder than many people might realize" in addition to getting a better knowledge of the product.

The net results of this process, though not easy, were that they had the ability to learn how to do any task on either of the teams. In other words, two teams with Generalizing Specialists (service mark Alan Page ;) ). In the process, each of the team members engagement increased, different perspectives were heard, learning and reflection were done in the retrospectives, and the teams learned to progress together. Perhaps the most valuable skill they both discovered they could do was to adapt to priorities and be able to pivot and change if/when necessary.

Don't Make Yourself Obsolete: an #STPCon Quasi Live Blog Entry

Design Inclusively: Future Proof Your Software

Since it will be impossible to have me post on my own talk, I'm going to give a pre-recorded recollection and thoughts about my talk and what I'm hoping to impart with it.

Accessibility deals with the ability to design software so that it can work with technologies to help people with various disabilities use applications that they otherwise would not be able to use. 

Inclusive Design allows programmers to create and design websites and applications that are available to the largest population possible without having to rely on external technology necessary for sites to be Accessible. 

Inclusive Design and Accessibility go hand in hand and are complementary endeavors but Inclusive Design, done early, can help make the last mile of Accessibility that much easier. That's the key takeaway I want to convince people to consider and advocate for. 

Inclusive Design is not magic. In many cases, it’s taking work that has already been done and making some small but significant changes. New web technologies help to make Inclusive Design more effective by utilizing semantic enhancements. More important, making this shift can also help you make better design choices in the future, without having to bolt on or re-architect your existing code, possibly at great cost in time, energy and finances. Sadly, we are still in a model where Accessibility/Inclusive Design is driven by two specific parameters:

- how much money do we stand to gain from doing this (because big deal pending and customer paying is demanding it)

- how much money do we stand to lose from not doing this (because we're actually being sued for being in violation of various disabilities acts)

Fact is, we can't really say what technology will be like in five, ten, or twenty years. We can, however, with great certainty, understand what we are likely to be like in those same time frames. When I talk about future proofing software. I don't mean from a technological factor, I mean in a usage factor. We're not future proofing for machines. We are future proofing for US! At some point, every one of us will leave the happy sphere of what is commonly called "normative". For some, it's never been a reality. For many, the cracks in that sphere start to appear around age 45. Seriously, I didn't care much about Accessibility or think much about it before I turned 45 and received the gift that keeps on giving (i.e. the need for reading glasses). That was my first step into the non-normative world of adaptive needs and being a target audience for Accessibility as an everyday part of life. I can assure you it will not be my last.

There are a variety of things that can be done and truth be told they do not have to be radical changes. Very often people will look at Accessibility and Inclusive Design changes and they will say "hey, wait a minute, we applied all of these changes and I don't see any difference." Right! That's the whole point. Accessibility and Inclusive Design doesn't have to be ugly or inelegant. I'd argue that Accessible and Inclusive software is actually more beautiful because its form is enhanced by its function.

Oh, and for those who have never seen my presentation, without spoiling the surprise, I'll share a phrase out of context that speaks volumes:

"IKEA GETS IT!!!"

Testers as Their Own Worst Enemies: an #STPcon Live Blog

Testers as Their Own Worst Enemies

Michael Bolton and I share a number of interests. We're both musicians and we have often talked about music and riffed with each other at conferences over the years. Michael starts out his talk with a clearly personal example. There's a, (rightly so, I believe) critical eye being placed on a variety of streaming services that are making a lot of money on technology to share music.

Those companies are leaving the key players of that product (i.e. the musicians) out of the equation, or at least are not compensating them in any way commensurate with their contribution. What happens when musicians are not ultimately compensated for their efforts and creativity? You either get less music or you get less quality music (a term that is hugely problematic and not something I will address; everyone has opinions on what good music is).

In many ways, software testers are feeling a bit like the musicians in the above example. Think about making music without musicians. Once that was considered unthinkable. Today, anyone with a laptop and some scripting skills *can* write a hit song (not that they necessarily will, but they absolutely can). What happens when we take musicians out of the equation of creating music? What do we lose? Likewise, what happens when we take testers out of the equation of making software? Some may argue that I'm stretching this analogy since programmers are closer to musicians than testers are but hey, my rules, I'm making them up as I go ;).

To torture this metaphor a little more, I want to make a plug to a person I think every tester should know. Truth be told, I do not know his real name. I only know him as "Blue".

Seriously, though, "Blue" is what I refer to as a "stand up philosopher" and in many ways, he's a great resource that I think any software tester will find value in both hearing and reading. Blue is young, like, I could be his Dad. Seriously. Still, Blue has a way of looking at the world and exploring how societies work that can be hugely helpful to a tester. He is also great at breaking down challenging topics and making them sound simple and I think this is the skill that every tester would benefit from. Seriously, check him out (end weird side tangent ;) ).

Testers need to be something other than just people who run tests. If that's all we bring to the table then we are ultimately expendable. We need to look at we actually do for a company. We test, sure, but if that's all we do, we are of limited value. If, however, we are capable of articulating what the issues are and why they would be positive or negative to the organization, using our brains and our persuasion, then we have some leverage. To that end, I will say that testers need to be stand up philosophers in our own right (see, I had a reason for pulling Blue into this conversation ;) ). When the dialogue about testers being social scientists comes up, this is what is meant by that. When we talk about risk, we need to humanize it. We need to make it relatable. We need to feel as though the issues affect us and others because, ultimately, they do.

Ultimately those of us that want to play in the testing game (for whatever reason) are going to have to make a case for the humanity that we provide. If we cannot or do not make the case for it, then we are effectively saying we are totally expendable. Testers need to stop looking at the areas where they can be farmed out or mechanized and draw attention to the areas that they really do provide value. Our mechanical output is not really that special. If it can be repeated, it can be automated, at least at the mechanical and procedural level. What can't be automated? Thoughtful synthesis and informed advocacy for doing the right thing and why our course of action would be right.

To borrow from my own advocacy area, I can talk about the fact that code can be checked for compliance when it comes to accessibility. Can I do that? Absolutely. can code do that? It sure can and probably a lot faster than I can. Can a system make an informed decision that the experience that a disabled person will have is comparable to one that a normative user will have? At this point in time, no machine can do that. You need people for that. People are good at advocacy. People are good at learning. People are good at making judgment calls. Testers would be well advised to place their efforts and emphasis on those humanities more so than the "Techne" of what they do.

QA/QE Supporting DevOps: an #STPCon Live Blog Entry

The QA/QE Role: Supporting DevOps the Smart Way

First off, Melissa Tondi is doing something I fully intend to steal. There are varying thoughts and approaches to having an introductory slide that introduces the speaker. Some don't use one at all. Some are required to do so at certain conferences. Melissa does something that I think is brilliant, funny and useful. Her first slide after the title simply starts with "Why Me?"

In short, Melissa is spelling out not who she is, or what her credentials are, but rather "you are here because you want to learn something. I want to give you the reasons why I think I'm the right person for that job here and now for you." Seriously, if you see me doing this at a future conference, props to Melissa and you saw it here first ;).



One of the avenues that Melissa encourages is the idea of re-tuning the methodologies that already exist. One aspect that I appreciate is Melissa's emphasis on not just QA (Quality Assurance) but also QE (Quality Engineering). They are often seen as being interchangeable, but the fact is they are not. They have distinctive roles and software testers frequently traverse both disciplines. The title is not as important as what is being done. Additionally, a key part of this is the ability to balance both technical acumen and user advocacy. In short, push yourself closer to Quality Engineering so that you can be an influence on the building of the software, even before the software gets built.

Introducing DevOps to an organization can be a wild ride since for so many people we don't even know what Dev Ops is. Melissa is using Anne Hungate's definition of "The collapse and automation of the software delivery supply chain". For many, that starts and ends with building the code, testing the code and deploying the code. The dream is a push button, where we press the button, everything is magic, and the software rolls out without any human interference. Sounds great and believe me, the closer we get to that, the better. We will step away from the fact that certain people won't be able to do that for practical business reasons but still having the ability in all of the key areas is of value.

There are some unique requirements in some countries and companies to have a title of "Engineer". That's a term that has a certain level of rigor associated with it and it's understandable that some would shy away from using an Engineering extension where it's not formally warranted. For this talk, let's set that aside and not consider QE as an official title but more as a mindset and a touch point for organizing principles. In short, you can be a QE in practice while still holding a QA title. Engineering presupposes that we are developing processes and implementing approaches to improve and refine work and systems.

On area that is definitely in the spotlight is test automation. A key point is that test automation does not make humans dispensible or expendable. It makes humans more efficient and able to focus on the important things. Automation helps remove busywork and that's a great place to apply it. Additionally, it's possible to automate stuff that nets little other than make screens flash and look pretty. Automating everything doesn't necessarily mean that we are automating important or intelligent items. Automation should get rid of the busy work so that testers can use their most important attribute (their brain) on the most important problems. Additionally, it's wise to get away from the "automate everything" mindset so that we are not making a monolithic monster that by its sheer weight and mass makes it unwieldy. By parallelizing or parameterizing tests, we can organize test scripts and test cases to be run when it's actually important to run them. In short, maybe it makes more sense to have "multiple runs" to come to a place of "multiple dones" rather than "run everything just because".

Use automation to help define what is shippable. There shouldn't be an after the fact focus on automating tests if they are actually important. By focusing on automation earlier in the process, you get some additional valuable add-ons, too. You limit the accrual of technical debt. You shake out issues with unscripted testing first. More to the point, you can address testability issues sooner (yes, I've mentioned this multiple times during this conference. I completed "30 Days of Testability" and now I have it clearly on the brain. Testability should be addressed early and it should be addressed often. the more testable your application, the more Automizeable the application will be (Oh Alan Richardson I have so fallen in love with that word ;) (LOL!) ).

What not to Test: an #STPCon Live Blog Entry

“Help, I’m Drowning in 2 Week Sprints, Please Tell Me What not to Test”

This talk title speaks to me at such a profound level. Be warned, I may veer into tangent territory here. There are what I cal "swells" that come with sprint madness and sprint fatigue. It's never constant, it's like a set of waves that you would look to time. For those familiar with surfing, this likely makes sense. For those not familiar, waves tend to group together and swells grow and shrink. These series of increasing and decreasing waves are referred to as "sets" and the goal is to time the set that feels good to you. Too early and you don't catch a wave. Too late and the wave wipes you out. In between is the rideable waves to drop in.

Sprints are the software mental metaphor that goes with "timing the waves" but the problem is that the timing of sprints is constant, but wave sets are not. Likewise, even figuring out what matters for a sprint may take more or less time any given sprint. Tasks like backlog grooming, story workshops, sprint planning, etc. all come down to making sure that we have an understanding of what matters and what's actually available to us.

Risk-based testing is the idea that we focus our attention on the areas that present the most potential danger and we work to mitigate that. We all know (or should know) that we can't get to everything. Thus we need to focus on the areas that really matter.

Mary recommends that we place emphasis on testing ideas. Testing ideas should go beyond the acceptance criteria. We can easily be swayed to think that focusing on the acceptance criteria is the best use of our time but often, we discover that with a little additional looking, we can find a variety of problems that simply looking at acceptance criteria won't cover. We also need to be aware that we can also range far afield, perhaps too far afield if we are not mindful. Test ideas are helpful but don't just play "what if" without asking the most basic question of "which would be the riskiest area if we weren't to address?"

An area that I find that happens (tangent time) is that I will be testing something and find that we have to deal with an issue related to our product ut has nothing to do with stories in play. As I am the owner of our CI/CD pipeline (note: that doesn't mean I'm the expert, just that I own it and I am the one responsible for it working properly). If something happens to our CI/CD pipeline, who do you think is the first person to spring into firefight mode? Are you guessing me? Congratulations! In a sprint, I don't have the luxury of saying "oh, sorry, I can't deal with pipeline issues, I have to finish testing these stories". Therefore, any time I have issues such as a pipeline problem that needs to be addressed, I immediately put a spike into the sprint. I do my best to consider how much time it will take and if I can handle it myself (often the case) or if I need to pull in development or ops resources (also often the case).  What happens over time is that we get a clearer picture of not just actual testing focus but also the legitimate interruptions that are real and necessary to deal with. In a sprint, there is a finite amount of time and attention any of us can spend. Time and attention spent on one area necessitate that it is not spent elsewhere and no, saying you'll stay up later to cover it is robbing your future self of effectiveness. If you are doing that, STOP IT!!!

Performing a test gap analysis is also helpful. In a perfect world, we have test cases, they've been defined and we have enough information to create automated tests around them as the functionality is coming together. Reality often proves to scuttle that ideal condition, or at least it means that we come up short a bunch. What we often discover is a range of technical debt. Areas may be well covered and easily documented with test cases and automated tests. Other areas may prove to be stubborn to this goal (it may be as simple as "this is an area where we need to spend some time to determine overall testability).

The Pareto Principle is a rule of thumb, it's not absolute. Still, the old adage that twenty percent of something is going to give you eighty percent of outcomes is remarkably resilient. That's why it's a rule of thumb in the first place.

Twenty percent of test ideas can help you find eight percent of the issues.
Twenty percent of the application features will be used by eighty percent of the customers.

What does this mean? It means you need to get a read on what's actually being used. Analytics and an understanding of them are essential. More important, using analytics on your test systems is important, not just the prod numbers. One thing that was driven home to me some time back was the fact that analytics need to be examined and the configurations need to be experimented with. Otherwise, yes, you can have analytics in place but do you actually know if you have them turned on in the right places? How would you know?

One more interesting avenue to consider is that you cannot test everything but you can come up with some interesting combinations. This is where the idea of all pairs or pairwise testing comes into play. Testers may be familiar with the all pairs terminology. It's basically an orthogonal array where you take a full matrix and from that matrix, you look at the unique pairs that can be created (some feature paired with some platform as an example). By looking for unique pairs, you can trim don a lot of the tests necessary. It's not perfect, and don't use it blindly. Some tests will require they be run for every supported platform and not doing so will be irresponsible. Still, prudent use of pairwise testing can be a huge help.

One Thing Testers Can Learn from Machine Learning: an #STPCon Live Blog Entry

Today's keynote is coming to us courtesy of Mary Thorn. Mary's been in the game about twenty years including a stint as a COBOL programmer relative to the Y2K changes that were happening at the time. Mary is also going to be doing a second talk a little later, so she's going to be in this blog twice today.

All right, let's get a show of hands. How many people have seen a marked increase in hearing the term Machine Learning? How many people feel that they understand what that is? It's OK if you are not completely sure. I feel less sure each time I see a talk on this topic. Let's start with a definition. Arthur Samuel defined it as: “Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed.”  The key here is that the machine can learn and can then execute actions based on that learning. This includes a couple of terms, supervised and unsupervised learning. Supervised learning is the process of learning something that maps an input to an output based on example input-output pairs. Each example is a pair consisting of an input and the desired output. Unsupervised learning groups unlabeled and unclassified data and by using cluster analysis identifies commonalities in the data and reacts based on the presence/absence of commonalities.

OK, that's fascinating but what does this actually mean? What it means is that our systems are not just the dumb input/output systems we tend to look at them as. There's a change in the skills we need to have the ways that we focus on our testing efforts. More to the point, there's a variety of newer skills that testers need to develop. It may not be necessary to learn all of them but testers (or at least test teams) would be well suited to develop levels of understanding around automation, performance, dev ops, exploratory testing, and pipeline CI/CD skills. Those skills may or may not reside in the same team member but they definitely need to reside in the team.

There are a couple of things that are helpful to make sure that this process leads to the best results.  The first is that there is a specific goal or set of goals to be pointing towards. In the process, we need to look at the outputs of our processes and examine what the data tells us and following it where it leads. Be sure, we may learn things we don't really want to know. Do we have defined tests for key areas? How are they being used? Do they matter in the first place? What are the interesting things that jump out to use? How can we help to determine if there is a cluster of issues? This is where exploratory testing can be a big help. Automation can help us consolidate the busywork and gather things together in areas. From there, we as individual testers can look at the data output and look for patterns. Additionally, and this is apropos to the 30 Dys of Testability focus I jumped through in March, we can use the data we have received and analyzed to help us determine the testability of an application. Once we determine areas where testability might be lacking, we should do what we can to emphasize and increase the overall testability of our applications.

Analytics are hugely helpful here. By examining the analytics we can look at areas where we can determine what platforms are used, what features actually matter, and what interactions are the most important. In short, let's make sure we learn what our app is actually doing, not just what we think or want them to do.

Getting underway at #STPCon Spring 2019

Welcome everyone.

We are just a little bit away from getting the conference started. The workshop days are over and with them my “quieter than normal streak. Today starts the multi blog entries that you either love or dread ;).

A couple of reminders. First, when I live blog it’s stream of consciousness. I may or may not have clean posts. Grammar may be dodgy. I may spin off into tangents. Some of my thoughts may come off half baked. I assure you this is normal :). 

I may go back and clean up entries for gross grammatical issues but other than that what you see is what you get. My impressions, mostly unfiltered. Hope you enjoy the series :).

Saab 99 GLE vs Mazda Miata MK1: Adventures in Car Restoration and Test Framework Building

Now that I have had a couple of times (and some dress rehearsals leading up to them) I feel pretty good about the material that I've been presenting in my workshop "How to Build a Testing Framework From Scratch". Actually, I need to take a small step back and say that the "From Scratch" part isn't really the truth. This workshop doesn't really "build" anything from the initial code level.

Instead, it deals with finding various components and piecing them together and with additional glue code in various places making everything work together. As a metaphor for this, I like to make a comparison to restoring cars. I'm not super mechanically inclined but like many people who were young and had more imagination than common sense, I harbored a desire to take an older car and "restore" it. My daughter has had a similar desire recently. Both of us have undertaken this process in our late teen/early 20s but both of us had dramatically different experiences.

When I was younger, I had the opportunity to pick up relatively cheaply a 1978 Saab 99 GLE. It looked a lot like this:

For those not familiar with Saab, it's a Swedish car brand that produced cars under that name from 1945 until 2012. It's a boutique brand, with a dedicated fan base. It has a few distinctive features, one of the entertaining ones being the fact that the ignition (at least for many of the vehicles) was on the floor between the driver seats. The key point is that it was not a vehicle where a large number of them were made. It's a rare bird and finding parts for rare birds can be a challenge. In some cases, I was not able to find original parts, so I had to pay for specialized aftermarket products and those were expensive. It also had a unique style of transmission that was really expensive to fix. Any guesses on one of the major projects I had to undertake with this car? The price tag for this was $3,000 and that was in 1987 dollars :(. When it ran, it was awesome. When it broke, it was a pricey thing to fix. Sadly, over the few years I had it, the number of days where it didn't work or needed work outweighed the days when it was working in a way that made me happy. I ultimately abandoned the project in 1990. There were just too many open-ended issues that were too hard or too expensive to fix.

By contrast, my daughter has embarked on her own adventure in car restoration. Her choice? A 1997 Mazda MX-5 Miata MK1. Her car looks a lot like this:

Her experience with "restoring" her vehicle and getting it to the condition she wants it to be, while not entirely cheap, has been a much less expensive proposition compared to my "Saab story" (hey, I had to put up with that pun for years, so you get to share with me ;) ). The reason? The Mazda Miata was and is a very popular car, what's more, a large number of them were made and they have a very devoted fan base. Because of that, Mazda Miata parts are relatively easy to find and there are a large number of companies that make aftermarket parts for them. With popularity and interest comes availability and access. Additionally, with a small size and relatively simple construction, there are a lot of areas that she can do work on the car herself that doesn't require specialized parts or tools. In short, her experiences are night and day different as compared to mine.

Have you stuck with me through my analogy? Excellent! Then the takeaway of this should be easy to appreciate. When we develop a testing framework, it may be tempting to go with something that is super new or has some specialized features that we fall in love with. There is a danger in loving something new or esoteric. There may or may not be expertise or support for the tools you may like or want to use. There may be a need to make something that doesn't currently exist. The more often that needs to be done, the more tied into your solution you are and will have to be. That may or may not be a plus. By contrast, using something that is more ubiquitous, something that has a lot of community support will be easier to implement and will also be easier to maintain and modify over time. It also allows for a greater flexibility to work with other applications where an esoteric or dedicated framework with exotic elements may not have it.

Stay tuned in future installments as I tell you why I chose to use Java, Maven, JUnit, and Cucumber-JVM to serve as the chassis for my testing framework example. Consider it my deciding I'd rather restore a Mazda Miata over a Saab 99 GLE.

The Second Arrow: an #STPCon Live-ish Blog Entry

Yesterday was the start of the workshops day at STP Con and I was happy to present for the second time "How to Build a Testing Framework From Scratch". It's done. I've had a chance to sleep on it after being emotionally spent from giving it. Now I can chat a bit about the experience and some lessons learned.

First, I was able to deliver the entire presentation in three hours, which blows my mind.

Second, I think the time spent talking about the reasoning behind why we might do certain things is every bit as important as the actual technical details.

Third, I've come to realize that there is an odd excitement/dread mix when presenting. Many people say that they are most nervous the first time they present a talk or presentation. I've decided I'm more nervous the second time I present something. The first time I may get through on beginner's luck or if I do take an arrow in the process (meaning I realize areas I messed up or could do better) that's in the moment and it's experienced, processed and put away for future reflection.

I use the term "arrow" specifically due to an old podcast where Merlin Mann represented this idea. Someone in battle feels the first arrow that hits them. It hurts, but it doesn't hurt nearly as much as the second arrow. The reason? The first arrow hits us by surprise. The second arrow we know is coming. It's the same impact but because I've been there and done that, I am often frustrated when the efforts to mitigate the issues I dealt with the first time aren't mitigated or something else I hadn't considered happens.

Much of this came down to making materials available to people in a way that was useful and timely. As I talked to a number of participants, we realized we had several similar problems:

- the materials were made available in advance but some people waited until the night before at the hotel to download them and discovered the hotel bandwidth couldn't handle it.

- the flash drive I handed off (though I did my best to make sure it was read/write on as many machines as possible) ended up as read-only on some machines. Thus it meant copying everything over to bring up the environment, which took close to a half hour for many people.

- even with all of this, I still managed to have to hear (more times than I wanted to), "sorry, my Hyper-V manager is set up by my company. I can't mount the flash drive or open the files". Ugh! On the "bright side" that was a situation that I couldn't control for or do anything about even if everything else worked flawlessly. Still, it was frustrating to have to tell so many people to buddy up with someone who could install everything.

So what did I learn taking my second arrow with this presentation?

1. The immediate install party will only ever happen if everyone in advance confirms that they are up and running well before the event. While the flash drives certainly help, they don't provide that large a time savings as compared to just having everyone set up when they walk in.

2. The "set up" and "rationale" part of my talk... since it's a workshop, what I should be doing (I think), is getting into the nuts and bolts immediately, and sharing rationale around each part of the process as we are getting into it. As it was, my introductory material took about 40 minutes to get through before we fired up the IDE and explored the framework itself. That's too long. Granted, it's there so that people can get everything installed but I think I can pace it better going forward.

3. Though the framework I offer is bare bones, I think I can comment better in the examples and should have some before and after examples that use different aspects and let people see them as a natural progression. Perhaps have three maven projects, each being a further progression from the last one.

Don't get me wrong, I had a blast giving this workshop and I hope the participants likewise enjoyed it. Still, I hope I can make it better going forward and here's hoping I'll get another chance to present it at another conference and hopefully not end up taking the third arrow ;).