PNSQC 2020 Live Blog: Generator-Based Testing: A State By State Approach with Chris Struble

Over the past several decades, we have seen a shift away from manual testing to automated test execution. While automation can help us take out a fair amount of the tedium of everyday testing, what we still deal with is test design at a manual level. Most of our tests have to be considered and have to be designed to be effective. Examples are good and testers are god at this but again, we are not terribly patient so we don't necessarily do a broad number of tests because, well, we are lazy.

  

What if, instead, we could generate test cases based on software coming up with more examples than we can provide? Testers create a description based on the tests they want to run. based on the data the tester provides, systems would then be able to generate additional test cases and generate multiple examples of tests we can run. I do some stuff like this currently and manage a number of for loops with a list of areas in want to focus on. that's a rudimentary level of this, but what Chris is describing goes considerably farther.

Our typical test case matrix focuses on Example-based testing (EBT). Test cases are manually designed one example at a time. we can then fill in parameters for those tests and then execute the tests we create. the problem is, of course, EBT can only test examples that are part of my matrix. It doesn't test for things I did not include.

Generator-based Testing (GBT) works by providing our descriptive tests to a system and that system can then take what we provide and create random tests that muck with the data or creates oddball examples (fuzz testing fits in this mold). 

I've had a dream since reading James Whittaker's book "How to Break Software" many years back. He has a first maxim that I have always loved, which is "look through the code and find every error message listed. Figure out what is meant to cause that error message to display, and then get that message to display through software interactions at least once." I'm paraphrasing but you get the idea. How cool would it be to be able to have a tool that would specifically look for error messages, determine where in the code those errors were supposed to display, and then give me a matrix of options related to automated tests that found be run to surface that error.

This is an interesting thought experiment and I confess not one I have given a lot of attention to.