Combinatorial Approach to Testing Mobile Applications: An Experimental Investigation Brandi Amstutz Advisor: Dr. Sergiy Vilkomir This research is supported by the National Science Foundation REU Site Award

Importance of Research Mobile testing is a critical step in the development process where errors in an application can be discovered and corrected to increase reliability and functionality. Traditional software testing strategies can be modified and implemented to solve the challenges associated with mobile device selection. The amount of devices required to exhaustively represent todays mobile market is extremely high due to a wide range of variability between available devices. This investigative research analyzes the efficiency of mobile testing based on a combinatorial approach to device selection. Introduction

Research Questions Addressed Which device characteristics have the potential to influence application reliability? Both hardware and software components have the potential to influence an applications functionality. Hardware components considered for this approach include: Mobile device type Screen resolution Hardware manufacturer Software attributes included: Android OS version RAM memory Introduction

Research Questions Addressed How should device selection be executed to ensure an optimal set of devices? This approach suggests parameter value specification based on user profile data reflecting the current mobile market. Less popular parameter values can be grouped to reduce the total possible values without compromising proportionality Introduction Figure 2. Device Manufacturer Distribution Figure 1. Screen Resolution Distribution Source: Unity 3D Mobile StatisticsUnity 3D Mobile Statistics

Research Questions Addressed How many devices are sufficient for mobile testing? Exhaustive testing strategies are limited by resource constraints and contain redundancy between device attributes. This research suggests a set of five devices selected based on device parameterization and combinatorial testing techniques for maximum coverage Introduction

Device Parameterization Based on user profile data we suggest to use the following device characteristics and their parameter values: Materials and Methods: Step One Type Device Manufacturer Android OS Version Screen Resolution RAM Memory SmartphoneHTCHoneycomb ( ) Low less than 426 dp x 320 dp < 768 MB TabletMotorola Ice Cream Sandwich (4.0) Medium at least 470 dp x 320 dp 768 MB SamsungJelly Bean ( ) High at least 640 dp x480 dp Other Extra High at least 960 dp x 720 dp Table 1. Selected Parameter Values

Device Selection Through Each-Choice coverage techniques and consideration of current market statistics we suggest the following two sets of selected devices: Materials and Methods: Step Two Based on user profile data the following devices were included due to their high level of popularity: Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0 Set B: Galaxy S2 i9100 and Galaxy Tab 10.1 All other devices have been added to provide total coverage of device characteristics Table 2. Selected Device Sets

Application Selection Three applications with similar functionality were selected for testing. The following applications used provide access to search, book and review travel accommodations ranging from car rental to discount vacation packages: Materials and Methods: Step Three Hotwire Travelocity BookIt.com

Device Access Perfecto Mobile cloud services were used for remote access to devices for testing purposes. Tests were recorded and screen shots were saved for later analysis. Remote testing procedure using SaaS cloud-based device access: Materials and Methods: Step Four User provides test input Input is sent to devices selected by user Audio & video of live test execution is recorded Test results & video are sent to user for analysis

Device Access Materials and Methods: Step Four Figure 3. Perfecto Mobile Dashboard

Additional Devices For comparison, two sets of random devices equivalent in size to the Each-Choice sets were selected. The same testing procedures were executed on the ten additional devices: Materials and Methods: Step Five Rando m Set Device ModelType Device Manufacturer Android OS Screen Resolution RAM Memory A Galaxy Note 10.1 TabletSamsungJelly BeanExtra High 768 MB HTC FirstSmartphoneHTCJelly BeanExtra High 768 MB Galaxy NexusSmartphoneSamsungIce Cream SandwichExtra High 768 MB Droid Razr MiniSmartphoneMotorolaJelly BeanHigh 768 MB Experia MiroSmartphoneOtherIce Cream SandwichMedium< 768 MB B Droid X2SmartphoneMotorolaOtherHigh 768 MB Nexus 7TabletOtherJelly BeanHigh 768 MB RezoundSmartphoneHTCIce Cream SandwichExtra High 768 MB Kindle Fire HDTabletOtherIce Cream SandwichExtra High 768 MB Moto XSmartphoneMotorolaJelly BeanExtra High 768 MB Table 3. Random Device Sets

Screenshot Analysis Screenshots were analyzed from all sets of devices for fault identification Results Figure 4 (left) illustrates Fault H5 for the Hotwire application. The expected result is that the number of rooms should update on the main page when a user selects a new number of desired hotel rooms from the drop down menu. The fault occurs when the room quantity is not updated appropriately

Screenshot Analysis Results Figure 5 (right) illustrates fault T1 for the Travelocity.com application. The expectation is that the Travelocity menu bar with their gnome icon is displayed at the top of the home screen. The fault occurs when the menu bar is missing from view.

Screenshot Analysis Results Expected Fault Figure 6 (left) illustrates fault B1 of the Bookit.com application. The expectation is that a menu bar with current query's information and reference number is displayed at the top of the screen. The fault occurs when this menu bar is missing from view.

Fault Detection Results Fault detection totals are as follows: Hotwire: 7 2 device independent 5 device dependent Travelocity: 7 1 device independent 6 device dependent Bookit.com: 4 1 device independent 3 device dependent

Fault Detection Results In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the results of random testing. In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to the results of random testing. Table 4. Fault Detection

Conclusions The quantity of application faults detected by the selected set of devices was equal to or greater than the quantity detected by the random set of devices. Additionally, the random set of devices did not yield a fault that the selected set was not able to identify. This suggests that the device attributes selected were suitable based on their expected influence on application reliability. Combinatorial testing strategies provided sufficient parameter value coverage with a test suite of five devices. This research suggests that this small set of devices is effective for mobile testing while also reasonable enough for resource constraints. Conclusion

Questions?