Testing all the World’s Apps - a #pnsqc Live Blog

In my world, I basically test one app. I test its appearance on the web and I test it as it appears on a variety of mobile devices. That's about it. By Contrast, Jason Arbon runs a company that tests lots of apps. Like, seriously, LOTS of apps!

Jason asks us what we would do if we had to test a huge number of apps. Would we take an approach to try to test all of them as unique projects? It's logical to think that we would but Jason points out that, no that's not needed. A lot of apps reuse components from SDKs and even when elements are unique, they are often used in a similar manner. What if you had to test all of the world's apps? How would you approach your testing differently?

Three major problems needed to be solved to test at this scale:

Reuse of test artifacts and test logic

By developing test approaches at as high of a level as possible, we have the ability to create test templates and methods for creating test artifacts in a reliable or at least close to uniform manner. Over time, there is a way to look for common components and make a series of methods to examine which places an element might be and how it might be interacted with. Chances are that many of the steps will be very close in implementation.


Reliable test execution

Once the patterns have been determined and the set of possible locations has been mapped, it is possible to create a test harness that will go through and load a variety of apps (or a suite of similar apps) to make sure that the apps can be tested. It may seem like magic, but it's really leveraging the benefit of reused and reusable patterns.

One challenge is that a lot of services introduce latency to testing over the Internet. By setting up queuing and routing of test cases, the cases that need to be run get the priority that they need.

Unique ways to correlate and report on the test results

The reporting structure that Jason shows includes the type of app, the page type, and the load time on average for each of the page. This allows for an interesting view of how their own app relates to or competes with other apps. Wild stuff, I must say :).