Making the Move to Continuous Testing: a #STPCon Live Blog Entry

Sorry for the gap in the 10:00 hour but I took advantage of the opportunity to talk with a vendor that we actually work with during that hour to discuss an issue we've been dealing with. Kudo to them for taking the time, letting me vent and demonstrate and figuring out next steps.

With that, I'm definitely interested in seeing what is happening with Continuous Testing. I first started reading about this topic almost ten years ago and it is interesting that it is still a hot button topic.

We do a lot with Continuous Integration (CI) where I am and have made strides with Continuous Delivery (CD). Now I'm curious as to the possibility of implementing Continuous Testing (CT).

Alissa Lydon works with Sauce Labs, and she is starting with getting a better understanding of what CT actually means. It doesn't mean that automated tests are running constantly (though automation is a part of the discussion). It does mean testing at every stage of development.

The first step to getting CT to have a fighting chance is that everyone has to be committed to quality, which means every level of the development process needs to think about testing and staying ahead of the curve. Think of having programmers and testers sitting together, pair navigating and pair programming/testing to help get testing principles and approaches into the process of writing software before line one of code even gets written. The key is that everyone needs to be on the same page and as early as possible.

Step two is testing at every stage of the development cycle. No code should pass through any stage of development without some level of testing being performed. This can range everywhere from requirement provocations to unit tests to integration tests to end-to-end tests, whatever makes sense at the given point in time.

Let's segue here and mention that continuous testing does not necessarily mean automated testing. Automation can make sense at a variety of points but you can have a mix of automated and live/manual tests and still be considered CT. There is also a balance of the levels of testing that will be done at any given time.

Next is being able to leverage the skills of the team so that automated testing can advance and get put in place. While Automation is not the sole criteria for CT, it is definitely important to be able to make CT work.  It will take time and attention and it will certainly take some development chops. Automation needs to be treated every bit as important as the production code and I'll fight anyone who says otherwise ;).

Additionally, a breadth of devices and platforms are critical to getting a realistic view of how your environment will look on a broad cross-section of resolutions and sizes, as well as user agents.

The ability to scale the systems being tested is important. Yes, we may start with just one machine, but ideally, we want to be able to simulate our production environment, even if we go with a smaller number of machines and try to extrapolate what a larger number would provide.

The final step is implementing and actualy using/reviewing analytics. In short, know what your product is doing and what is being used to focus efforts.