Mohit Anand, Software Engineer Adobe 1 Selecting GUI Automation Testing Tool for Mobile Domain

2 Abstract With the growing number of mobile apps and their agile development life cycle, Automation testing around them will no longer be optional. The intent of this paper is to present a comparison between the widely used mobile automation tools in order to determine their ease of use & effectiveness. We picked up a typical Adobe mobile app, tested its GUI manually & then followed it up by a round of automation testing using these tools. This paper list down the technical nuances involved while choosing any such tool for user specific scenario.

3 Do you know your apps?

4 Complexities in Mobile Landscape Operating System Screen size Screen resolution Device Manufacturers Smartphone or Tablet GPU used

5 Operating System & Device Manufacturers

6 Screen Size & Resolution

7 GPU: Heart of Creative Apps

8 Localization: The Big hurdle Mobile apps stores are global, Mobile apps should be too. Supporting the app in around 18 locales Globalization ready

9 Our main consideration GUI Testing Functional Testing App success Usability Appealing Design Positive interaction

10 Need of Automation in Mobile Testing Growing mobile apps Agile development cycle Mobile Landscape Complexities Globalization: Apps releasing in multiple languages Automation is the key 7(Q1’14) – 40(Q1’16)

11 Mobile Automation Tools

12 Selection criteria We want to have a combination of tools which satisfy the following criteria:  Can automate Android/iOS/Both mobile apps  First party/Third party tool  Bundled with the OS IDE/ Should be installed separately  Coordinate based/ Object based automation

13 Tools Selected

14 Key Features Components of this tool Google Provides an API for writing programs to control an Android device/emulator. Can be used for both functional and UI testing. Entirely based on coordinates based automation. MonkeyRunner MonkeyDevice MonkeyImage MonkeyRunner

15 Strengths Easy to set-up. Test multiple devices at a time. Appropriate for regression testing of a basic app. Random testing. Limitations Only supports Android. Different script for different device. Even slight change in UI breaks the script. MonkeyRunner

16 Key Features Components of this tool Google It is a java library containing API’s to create customized functional UI tests. Integrated with Google’s own Android development IDE Entirely based on object based automation. Uiautomatorviewer API framework. UI Automator

17 Strengths One of the best tools for testing android apps. Comes bundled with Android Studio IDE. Appropriate for UI testing of any android app. Limitations Only supports Android (versions > 4.3). Lack of support for Web view. Lack methods for some gestures like long press on an element. UI Automator

18 Key Features Components of this tool It is an integrated environment for recording, customizing and managing test suites. Open source, eclipse based IDE. Used for automation of both iOS & Android apps. Limited features are available in the free version of this tool. Paid version is costly. MonkeyTalk IDE MonkeyTalk Agent Cloud Monkey

19 Strengths Supports both iOS and Android apps. Test multiple devices at a time.Open sourceImage comparison is supported Limitations The essential features are available in the paid version. Doesn’t support web view. Agent needs to be installed in the app which needs to be tested. Limited xPath support.

20 Key Features Components of this tool Appium Server Appium Client libraries Appium Inspector It is a HTTP server that takes HTTP commands from the test script and executes the command on the device. Based on WebDriver JSON protocol. Used for automation of both iOS & Android apps. Totally derived from selenium code + mobile gesture handling Sauce Labs

21 Strengths Supports both iOS and Android apps. One of the most versatile tools for testing any native, hybrid and mobile web apps. Doesn’t require code access or recompiling the app. Test multiple android devices at a time. Limitations No support for parallel test for iOS apps. No support for Android version < 4.1 (selendroid can be used) No Image comparision. No option to switch off network connection.

22 Deep dive into Appium

23 Test Setup

24 The Process: Recipe of Success Know your mobile app. Identify the right automation tool. Write test script to cover all the testing workflows in the app. Run the script for the test pass. Next Release: Update the script. Repeat until the app lives on the app store.

25 Demo of Adobe Shape GUI Automation

26 Takeaways  Use real devices: In Adobe, we say no to simulators.  All elements in the UI should have a unique resource id.  The App UI has changed: Get ready for modifying your test script.  Say NO to sleeps and xPath.  iOS Automation: You need code access or debug build.

RIGHT TOOL Preference to free/paid tool Which app to automate: Android/iOS or both? Time Crunch Flexibility of tester with the scripting languages Community support First party/Third Party 27 Choose the RIGHT tool

28 Use case scenario Monkey Runner Android app onlyBasic UI testing Time Crunch Low effort/investment UI Automator Android app only Intensive UI testing. Don’t want to use third party tool. Have time to build a stable automation solution. Monkey Talk Android & iOS app Your app doesn’t supports web view. Image comparison requirement. Can install agents in your app / can invest money per month Appium Android & iOS app Intensive UI testing Don’t have code / Don’t want to install any agent. Stable automation solution (free) for suite of apps

29 Bibliography o o o o o

30 Question and Answers

31