1. Mobile Automation: Touching the Mobile World through code
Mobile Automation: Touching the Mobile World through code
Jason Edstrom

2. Mobile Automation Overview
Developing software to automate UI interactions with a mobile application
With so many device configurations to account for, Mobile Automation provides an approach to efficiently test different configurations
Still maturing, but new developments happen fast
As the industry matured, hosting companies started to fill the gap by supplying emulators and real devices to automate as a service
So what is Mobile Automation? It is developing software to automate UI interactions with a mobile application or browser. Due to the level of device fragmentation in the mobile market, this basic statement holds more weight than we give it credit. As a manual mobile tester, the work load and scenarios that need to be tested and managed can reach overwhelming levels. Between OS version, device specifications, app state, data state and any other state that you can think of, there is enough repetitive testing opportunities that would be good to send to automation. Mobile automation is an efficient way to get through all of those configuration scenarios. The mobile automation category is consistently changing. The tech stack state from 6-12 months ago is different than today. New developments evolve to solutions that can completely overcome large system roadblocks. As the industry has grown, the need for device inventory has expanded along with it. So the gap for device configurations is now starting to be filled by companies providing a service like Perfecto Mobile and Sauce Labs.

3. Types of Tools 3 main categories of mobile automation implementation
External
Internal
Image Based Recognition / Optical Character Recognition (OCR)
There are a number of companies that implement mobile automation and with that come different capabilities and requirements. When they are boiled down to the fundamentals, there are three main categories of mobile automation implementation. External, Internal, and Image Based Recognition / OCR.

4. External
Uses the internal native libraries to allow automation via an external server
No need to have access to the code base
Predictable and well accepted by the community
Tools
Appium
Selenium
The first implementation is external. The driving component is an external server that links against internal native libraries to allow automation. Android and iOS both have JavaScript automation frameworks under the hood. Android is UIAutomator, iOS is UIAutomation. External implementation tends to be the less invasive route which affords us to not have access to the code base. An build deployed to hockeyapp will work just as well as having the code base with you. The tools that use this approach are well accepted and tested by the community. This allows for trouble shooting to be predictable and fairly well documented.

5. Internal
Uses a vendor library that is compiled into the app to allow the tool to see and interaction with UI elements
Need access to the code base to implement the automation libraries
Remember to remove the library before production builds
Tools
Telerik
Ranorex
Monkey Talk
TestComplete
The next level is the internal implementation. With internal, there needs to be a library compiled into the app to allow the tool to see and interact with the UI Elements. With this scenario, you either need access to the code base in some way or having buy in from the client to add the library. You also need to work against any compatibility issues with the host app’s SDK version. Since the need for a library, developers also need to ensure that the library is remove at the time of submission to the respective app store. The security issue that the library presents will cause the application to be rejected during the submission review process. You can also be bound to the language of choice from the software provider. The nice detail about the library is that it can offer some unique approaches to testing. Telerik uses a test cloud portal platform where you write your tests in a JavaScript variant and the tests are transmitted to the device. It can also launch the application under test inside of their app.

6. Image Based / Optical Character Recognition
The “Hail Mary Pass” of mobile automation
Unstable and Unpredictable
Dependent on assets loading in a timely fashion to continue interaction
Tools
SeeTest
Sikuli
Imaged Based and OCR tends to be the last ditch option for mobile automation. Whether you have compatibility with SDKs or development frameworks, if you are unable to recognize objects, this is usually your last option. It can be unstable and unpredictable to a level where the best tools using this tech are decent at best. Your success of automation tends to be dependent on assets loading in a timely fashion to continue the automation interaction and the validations.

7. Considerations Not all mobile apps are created equal
Frameworks
Xamarin
Cordova
Flex Mobile
SDK Versions
Enterprise Architecture
Mobile Device Management
Mobile Application Management
XenMobile
AirWatch
Inventory
Poor QA process scenarios
Some worthwhile considerations before jumping into mobile automation. Not all apps are created equal. Just because Company A developed App B, it doesn’t mean that you can automatically automate Apps C, D, and E. At any point, new development frameworks can be added and sdk requirements. Some frameworks handle the translation in stellar form. A good example is Xamarin with getting as close on the native ui components as you can get without the development being native code. Other frameworks handle it well enough with a few obstacles like Cordova. Then, if you are unfortunate enough, you can get a framework like Flex Mobile… where you are luck to be able to translate a Text Field to a native component. There are enterprise architectures that are designed to stop the level of system interaction that we are working towards, both on the application level and device level. Another large consideration is the device inventory. If there is a large supported device and os library, your testing scope just got massive. The last consideration is poor process scenarios that need to be changed. A good example is XenMobile mentioned above. A poor process scenario is that testing happens after the app is wrapped in the security layer. XenMobile tends to introduce new bugs to the application so you don’t know if the bug is the MAM wrapper or the development. Second, it blocks the potential for automation by abstracting away the automation frameworks housed in the OS.

8. Appium Philosophy
Appium was designed to meet mobile automation needs according to a philosophy outlined by the following four tenets:
You shouldn’t have to recompile your app or modify it in any way in order to automate it.
You shouldn’t be locked into a specific language or framework to write and run your tests.
A mobile automation framework shouldn’t reinvent the wheel when it comes to automation APIs.
A mobile automation framework should be open source, in spirit and practice as well as in name!
So that we laid the ground work for what the mobile automation frontier currently looks like. I want to shift our focus to a specific tool called Appium. I listed it under the external tools category and they do a wonderful job of justifying that position in their philosophy. It can be summed in the phrase of “lazy freedom”. They believe that a good tool shouldn’t force you to modify the application, be locked into a specific language, create a whole new automation api set and then pay out the nose of it.

9. Appium Strengths Uses a client/server architecture approach
An external webserver instance written in Node.js
Different drivers depending on your language and OS preferences
Real Device and Emulator
Uses Mobile Element objects for validation and navigation
Cross-Platform (iOS & Android)
Supports most major languages
Java
PHP
Objective-C
Python
Ruby
JavaScript C#
Others
Supports Native, Web and Hybrid apps
Large Support Community and consistently evolving
Predictable and relatively easy to troubleshoot
It is capable of automating against Real Devices and Emulators. You have access to cross platform automation against the 2 major mobile Oses. You have access to any language that there is a selenium driver for. You get the coverage of the 3 types of mobile applications that can be developed. You have the large support community that is consistently contributing to the quality of the product and its troubleshooting.

10. Appium / Selenium Uses the same JSON Wire Protocol as Selenium
Appium driver and its children, iOS Driver and Android Driver, are just extensions of Selenium’s Remote Web Driver
Enhanced with Mobile related interactions
Tap
Swipe
Long Press
Shake
App Installation and Close
Multi-Finger
Etc.
The differences are more so related to the server and its setup
With Appium, you can take a large number of the selenium web automation techniques and concepts and apply them to mobile automation. It makes the transition from Web Automation to Mobile Automation close to seamless from a development perspective. It just takes a little bit of a mental shift to mobile thinking. Under the hood, it uses the same JSON wire protocol that selenium uses. Appium driver and the children are actually just extensions of the Remote Web Driver. You can write mobile tests using the Remote Web Driver, but you lose all of the mobile goodness that comes with the driver. With the similarities to selenium, we are able to use page object design patterns for recognition and validation along with parallel test runs with the use of GRID. The major differences comes from the side of the server and the setup.

11. Appium Requirements
Appium is rather strict on configuration parameters in order to coordinate between all of the devices.
iOS
App that is signed as development build
AirSign and iReSign
OS X is needed to execute the tests and inspect the app, but not required the write the automation code
Xcode Instruments is what drives the native UIAutomation library.
Android
If apk is on hand, Appium GUI can auto populate most of the needed fields.
If apk is no available, then the app needs to be decompiled to find the needed info.
Where Appium can get difficult is the requirements for the applications under test. iOS is significantly different from Android. For iOS, the app package needs to be signed as a development build. It doesn’t need to start out that way, but it needs to be in that state at the point of test execution. Xcode provides tooling to resign code as long as you have a developer certificate and provisioning profile. I have used AirSign to check my entitlements, permissions and signing. iResign is a github project that makes it easy to use the Xcode tooling for resigning code. Way more enjoyable than figuring out the command line every single time. The second iOS detail is that you need a mac as a server node in order to execute and inspect the app. The test code can be written anywhere, but the device need to be connected to a mac to run. Android is far more simple. It is highly perferable to have the APK, but that is about all you need. The app installed somewhere or the application package.

12. Appium Environments Device Farms
Perfecto Mobile
Able to be used in a GRID configuration
Emulation
Sauce Labs
Along with the server implementation, there is a standalone app
Windows -> v
OSX -> v (with the option to upgrade to 1.5.2)
Server -> v 1.5.2
Appium test environments are growing. You have Perfecto which advertises their Appium integration. You can take a locally developed test and easily direct it to their service to start automating tests. It requires changing the web address that the automation is pointed to along with some of the device configuration info, but after that you have the same code base. You can build out an on premise GRID device farms in the same approach. You can use emulation, either locally or through a service. I typically use the Server implementation for the mass test runs, but I favor the standalone app for my development and troubleshooting. There has been a recent major code refactor on the server where most of the server configuration is code driven. The standalone apps are starting to get the same attention like on OS X, but they have some bug fixing to go through. The standalone has a large feature difference between Windows and OS X versions.

13. Appium GUI: Windows
Windows Appium is only able to work on Android because the OS X development tools are not available for windows. It had a long period where it didn’t receive updates to the internal node server because it is a half implemented solution. With Appium, you get a tool called Inspector. It parses the page source xml from the mobile UI and displays in a tree format with all the node attributes visible.

14. Appium GUI: Windows
Unfortunately, Inspector on windows Appium is also lacking. You get basic information as is. Xpaths are not generated and you can only mimic 2 driver interactions from the inspector

15. Appium GUI: OS X
OS X is significantly more flexible. It feels weird saying that about something on apple, but it is what it is. I have the option of both iOS and Android platforms along with GRID configurations, web hooks and a ton of other goodies. Also you get Appium doctor which helps with any Appium setup troubleshooting that you need to handle.

16. Appium GUI: OS X
Appium inspector on OS X is also better by leaps and bounds. I get all the attributes plus Xpath and UI hierarchy along with more driver interactions and a selector tool to help check my elements. It is much easier to use and more powerful. I recommend giving this version a try… and if give the windows one to someone as a learning tool.

17. Mobile Selectors: Android
This is a screenshot of an android app called Criminal Intent. It is pretty much tattle tail app for the office. This is the UI layout code that is needed to build one of the screen in the app.

18. Mobile Selectors: Android
Selectors come directly from the layout XMLs that define the app’s UI.
Resource IDs provide unique element identification
Examples
Editable Text Field
Checkbox
In Android, the mobile selectors come directly from that layout XML. Resource IDs are

19. Mobile Selectors: iOS iOS is not as direct as Android
UI Automation JavaScript framework defines the element type
UI Automation framework taps into the Accessibility framework
Same framework that is used by Xcode Instruments to do automation.
ationRef/
True unique identifiers require specific UI element attribute changes from the developer.

20. Mobile Selectors: iOS

21. Mobile Selectors: iOS var target = UIATarget.localTarget();
target.frontMostApp().mainWindow().textViews()[0].tapWithOptions({tapOffset:{x:0.58, y:0.98}}).tap();
target.frontMostApp().keyboard().typeString(“The Sun is out! It feels great!");
IOSElement element = this.appiumDriver.FindElement(By.Xpath(“//*/UIATextView[1]”));
element.SendKeys(“The Sun is out! It feels great!");
DiaryEntryPage diaryEntry = new DiaryEntryPage();
diaryEntry.setDiaryEntryBodyText(“The Sun is out! It feels great!”);

22. Quirks and Nuances Android and iOS element attributes are different
Each OS has element attributes that the other does not which leads to different paths to find the element.
App Switching
Currently can’t automate outside of the target app’s “sandbox” in iOS
Android has issues with this as well, but for different reasons.
App Switching breaks the automation session and stops the automation
Recent developments could lead to this no longer being an issue

23. Mobile Browser Automation
Supports Mobile Chrome and Safari
Most desktop automation test should be transferrable to a mobile browser automation stack

24. Mobile Browser Automation: Android
Setup is easier due to how Android permissions system level apps
Standard Appium device configuration
Add the browser name parameter (Chrome or Internet)
Uses a variation of chrome driver
Node.js wrapper

25. Mobile Browser Automation: iOS
Currently has a number of requirements and moving parts
iOS does not allow Appium to open system level apps
We don’t have developer level ownership
Settings
Calculator
Calendar
Safari Launcher is required
Developer-signed
Since there is not an Appium wrapper safari driver, we need to get creative.
iOS WebKit Debug Proxy

26. Reinforcements! Mobile Automation paired with Services automation
Gives the ability to generate and manipulate data, then test the application to see if the changes are properly reflected
Regression and Smoke Tests
Let a machine do the repetitive stuff
System stability check
Unexpected scenario/data changes

27. Short Video Quick proof of capability with Selenium GRID Tests
Invalid Login
Login portal loads
Successful login
4 iOS Devices
iPhone 5s – iOS 8.1.4
iPhone 6s – iOS 9.0.2
iPhone 6 Plus – iOS 8.0.4
iPhone 6s Plus – iOS 9.3.2
App State managed by uninstalling and reinstalling the app on each test execution

28. Short Video Speedup video for presentation purposes
Resolution downgraded to support upload to TCTAG