2. WEBINAR Device Labs Boost Mobile Test Automation
John M. Wargo, Principal Analyst
May 11, Call in at 10:55 a.m. Eastern time

4. Agenda
Mobile App testing
Adding automation to the mix
The rise of the device lab
Next steps

5. Mobile app testing
Testing is hard

6. Many cycles with a variety of tools and processes
Unit testing
API testing
Functional testing
Performance testing
Acceptance testing
Developer workstation
Back-end
On-device
On-device and back-end
Manual

7. Old-school mobile testing means fingers poking at screens

8. Devices are required in multiple parts of the development process
Device access:
Developer tests on local devices (connected to the development workstation).
QA performs manual or exploratory tests on a set of devices.
Functional testing on as many devices as possible.
Few devices have cellular network connectivity.
Device sharing is a nightmare to manage, never seem to have enough devices or the right ones.

9. Manual testing drives a feedback-driven approach for testing edge cases
“As complexity of mobile applications and the testing process increases, focus efforts on key features of the app. Create a feedback loop with application users to identify issues with less-used features and fix them in the next release if possible.”
Source: Forrester Report: “Improving Mobile App Quality Testing”

10. What users do with bad apps

11. Angry users provide more feedback

12. Adding automation to the mix

13. The agile process both complicates and simplifies app testing
Developers more involved with testing
QA has a different role
Testing never ends

14. Automation is the key to mobile app testing success in an agile world.
Image source: Fotolia: (

15. The rise of device labs

16. Mobile testing research focuses on automation

17. On-premises or in the cloud

18. Devices are a key component of mobile app test automation

19. Device labs can be simple…
Image source: CrossBrowserTesting
Image source: SmartBear

20. Device labs can be sophisticated…
Image source: Dynatrace Corportation
Image source: Dynatrace

21. Many cycles with a variety of tools and processes
Unit testing
API testing
Functional testing
Performance testing
Acceptance testing
Developer workstation
Back-end
On-device
On-device and back-end
Manual

22. Critical part of a complete test suite

23. Both functional and performance testing

24. Real-world functional and performance testing

25. Next steps

26. Don’t rely upon emulators/simulators for native app testing
Less expensive than actual devices
Not the same as physical devices
Suitable for functional testing; unacceptable for performance or load testing

27. Cloud or on-premises? Constraint Cloud On-premises Multiple devices
Easy
Hard
Multiple OS
OS updates
Wi-Fi connectivity
Cellular connectivity
Easy, but $$
Secure environment
Possible*
*Use secure tunnel to the cloud or a hybrid deployment.

28. Get the most for your buck
Large catalog of devices for the most complete test coverage
Open source framework support
Appium and Selenium lead.
Calabash, Espresso, and Robotium are popular.
Parallel execution is critical for large apps, big test suites, and lots of devices.
Unless you have steep security requirements, let someone else manage your lab for you.

29. Economic impact is changing

30. Labs deliver results beyond automation
Select a solution that supports manual and exploratory testing
Streamline problem analysis through live debugging

31. Vary tests for best results
Add monkey tests to your test suites.
Test apps under device load.
Test apps under varying conditions.

32. Questions?

33. John M. Wargo