Parallelization and JavaScript: SF Selenium Meetup, Oct. 2013

Hey all, TESTHEAD is live at the San Francisco Selenium Meetup, and looking forward to another night of pizza, ginger beer and talk about testing in Parallel and testing JavaScript. What are YOU doing tonight ;)?


The following are from the Meetup site, so consider this pre-amble. You'll definitely get plenty of my ramble in a bit.


Optimization through Parallelization (David Borin)

Dramatically improve the scale and speed of testing through the use of parallelization techniques. This includes writing test modules that can be run independently, doing as much preloading of test data as possible, and using parallel test runners.

Cross Browser Compatibility Testing JavaScript (Alan Parkinson)

People often run their WebDriver based functional test suites across multiple browser combinations with the aim to test cross-browser compatibility. Functional tests provide poor coverage of JavaScript and long test execution times. I will demonstrate how you can reuse your existing JavaScript unit tests with Karma and utilizing your Selenium WebDriver test infrastructure for providing fast cross-browser JavaScript compatibility tests. If you're not already doing JavaScript TDD then the additional benefits highlighted in this talk will encourage you!

My thanks to Lithium Technologies for providing the space an the food. Speaking of food, I'm going to go get some now. Be back in a little bit.

Alan Parkinson is first out the gate with a talk about how to parallelize tests for JavaScript. The app that Alan's company creates is, by his own admission, mostly JavaScript. A key truth that Alan wants to get out to everyone... You don't need to run your Selenium Test suite with every browser (*). What?! Heresy, you say! Well, whats the reason we are testing against all the browsers? Is it rendering compatibility? Is it compatibility with the DOM? Is it to get lots of javaScript code coverage? According to Alan, no, there was about two thirds of their code that wasn't even being exercised. So how do we do this? According to Alan, the answer is is to create unit tests for your JavaScript... and run *THOSE* tests cross browser. Hmmm.... OK, I can see the logic to that.

Alan makes an interesting point when he describes the options between functional tests and JavaSctipt unit tests. An average functional test suite takes about five minutes to run. The JS unit tests... take about 11 seconds. Alan did mention that he had a qualifier, and here it is... not everything is going to be picked up by unit tests. There's far too many JavaScript files, and there's actual behavior aspects that unit tests will not actually be able to pick up. So some tests will have to be run cross-browser, but the point is, not as much as we might think.

So what test runner do they use? Alan talked a bit about Karma and their "Spectatular Test Runner for JavaScript". There are some challenges, to be sure, so to solve some of their challenges, the made a Karma WebDriver Launcher, which allowed them to leverage their Selenium WebDriver infrastructure. Alan took  a time out to run through Karma and show how they have set up their framework and which options to call. Granted, it's a small set of tests for the demo, but no doubt about it, it is remarkably fast. Running the tests in parallel on multiple browsers becomes a quick option compared to native Selenium WebDriver functional tests.

Karma provides code coverage, but to do that, you need to instrument your JavaScript (in their example, they do this with Istanbul). They can also generate code coverage reporting formats and the ability to come in and see what is actually covered and where.

Overall, the main takeaway that alan wanted to deliver was that functional tests are an expensive way to get browser compatibility coverage. By comparison, JavaScript unit tests can provide much more coverage, much quicker, and the beauty of it is... the programmers do the lions' share of the work with this model, since they would be doing it anyway. Again, interesting approach, and a clever approach to cross-browser testing (and to be clear, they do still run a number of functional tests cross-browser, but the total number of tests is significantly less, since they prefer letting the unit test framework go for it with the code coverage and atomic level functionality. Again, interesting idea, definitely want to explore the idea more :).
  

OK, next talk is up, and now it's time to talk a bit more about parallelization... imagine being able to run ten thousand test modules in under five minutes. The simple answer is that parallelization is necessary to get those numbers. Spin up a battery of virtual machines, spread the tests equally over the machine battery, and get your CI server to do all the work (living in a world that uses Jenkins and AWS, I can confirm that this method is real and does work.

This approach does require that the effort be made to make tests as atomic as possible, and that dependencies are, if not completely eliminated, at least minimized as much as possible. Also, you don't need hundreds of machines. With 4 servers, you can create a Jenkins master, a slave with 10 executors, Selenium Grid hub and selenium grid nodes with 15 browsers. I'll leave it to you to do the multiplication on that. The point is, with that setup, you can do a fair amount of parallel testing.

Several great takeaways for the talk were encapsulated in the "mantras" slide, so rather than repeat them, hey, it's right here :):

Tools for parallelization?

Wow, that went quickly! Glad to have the opportunity to come out and play once again. Lots of interesting ideas, and I definitely want to take a closer look at some of the ideas from these presentations. If I had to pick any one thing to consider, it would be to see how many tests are "overloaded" or have a lot of steps. Could those tests be puled apart? Would it make sense to run them separately (and subsequently, more in parallel)?

And with that, I bid all a good night. Happy testing :)!!!