In order to further explore GUI testing techniques discussed in a previous work, the authors of this study performed an experiment on two specific techniques for testing cellular phone applications. In particular, the authors endeavored to apply a formal experiment and statistical analysis to the speed and efficacy of DrivenHoper (DH) and their own Behavior Explorer Tool (BxT) GUI testing frameworks for Motorola phone apps. The two techniques are fairly similar; both involve bombarding a randomly selected starting "screen" from the application with randomly selected keypad input, then check for a system crash. The key difference between the two is that DH selects random keystrokes from the set of all possible commands, whereas BxT selects keystrokes only from the keypad events supported for the current screen.
The amount of overhead required to perform individual runs of these systems is significant; if the tests run to their completion without creating a crash, the total run time is a full 40 hours. With this in mind, the authors used a limited set of four phone hardware / software configurations found to produce the most variability in their previous study. The four phones were then each run through 8 trials each for the two testing techniques, with each trial representing one possible combination of the three inputs taken by the testing applications (re-initialization to an initial state mid test or none; use of weights for the selection of input values on a given screen or true random selection; and test case size of 50 vs. 500 key strokes per selected screen).
In all, DH successfully created a crash in 56.25% of its runs, whereas BxT created a crash in 87.5% of its runs, suggesting that BxT is more effective at identifying UI vulnerabilities. On average, BxT also proved faster at locating errors than DH in all trials. The various input values to the testing frameworks also had limited effect on this speed, particularly the combination of re-initialization and sample size.
Though BxT has proven comparatively successful in this study, the authors seem eager to expand upon and optimize the algorithm further. In the long term, the BxT technique may even be extended to software outside of the mobile phone domain.