1. Automated Pattern Based Mobile Testing
By Spencer Gray

2. Overview Motivation Background Supporting Paper’s Approach
Main Paper’s Approach
Conclusions

3. Motivation Our code never works on the first try
World Quality Reports show an increase in organizations that perform mobile application testing
~92% of organizations reported heavily testing their mobile applications
Testing is expensive
Testing all possible use cases is difficult

4. Background
By 2013 we passed the 1 million application mark on both Google Play and Apple’s App Store
This number has only continued to grow
Business critical and mobile banking require applications to be flawless
Goal – Test our software while we sleep
We want automated tests to identify as many bugs as possible
Nobody likes writing / doing tests!

5. Background - Common Approaches
Adapt established web and desktop approaches to mobile applications
Strengths
Don’t need to reinvent the wheel
Established methods are effective on their respective domains
Weaknesses
Event sequences are longer in mobile applications
Event handlers are smaller
Create new approaches… (explained later)

6. Background - Common Approaches
Random based testing (crawling)
Typically misses important functionality
Good algorithms can cover most cases (but not all)
Symbolic Execution
Based entirely on abstract model of application
Obvious flaw that it never runs the actual application code
Only looks at basic event handlers

7. Automated Testing with Targeted Event Sequence Generation
Supporting paper – published in 2013
Propose a targeted approach to generate event sequences
Given a target, work backwards to generate events to generate this target
Reverse engineer the application

8. Automated Testing with Targeted Event Sequence Generation
Motivated by the line reachability problem for C programs
Works backwards in the call graph from a line in a C program
Works until a feasible path from the beginning of the program is found

9. Automated Testing with Targeted Event Sequence Generation
In mobile applications we don’t have simple call graph
C programs control execution flow
Mobile apps are controlled by user
Solved by the first step of their two step process
Operates in a two step process:
1) Preprocess the application
Perform concolic execution for each event handler
Infer path conditions and symbolic states for all possible paths
Summary for each event handler

10. Automated Testing with Targeted Event Sequence Generation
Operates in a two step process:
2) Build concrete events sequences from entry point in application to a given target
Worklist algorithm executed on items consisting of a path ending at the target
Paths are extended by considering event handlers that can satisfy constraints in respective path conditions (determined in step 1)
Works backwards from the target to the entry

11. Automated Testing with Targeted Event Sequence Generation
Results
System was successful for many targets (not all)
Imprecise prototype caused most failures
Concolic execution (step 1) needs to be improved
Run time was ~30 minutes for automated portion

12. The iMPAcT Tool for Android Testing
Main paper – published in 2016
Presents the approach and tool called iMPAcT (“Mobile PAttern Testing”)
Focuses on pattern based testing
Defines pattern as “recurring solution for recurring problem”
Patterns are ubiquitous in mobile applications
Login screens
Navigation bars
Icons
Buttons

13. The iMPAcT Tool for Android Testing
iMPAct Tool guided by 4 main characteristics:
1) Reverse engineering process does not need the source code
Improvement over the supporting paper
2) Focus on testing recurring behavior
Behaviors to be tested can easily be added/removed as needed
3) Completely automatic process
4) Combines automatic exploration, reverse engineering, and testing

14. The iMPAcT Tool for Android Testing
Patterns are defined formally in the tuple <Goal, V, A, C P>
Goal – ID of the pattern. What are we trying to accomplish?
V – Set of key-value pairs relating input data to variables
A – Sequence of actions to perform
C – Checks to perform. Does the test pass?
P – Preconditions that need to be satisfied to apply the pattern
Context of the pattern

15. The iMPAcT Tool for Android Testing
Uses two types of patterns:
1) UI Patterns
Directly related to the application interface
Action Bar, Side Drawer, Login form, etc.
2) Test Pattern
Test strategy for a corresponding UI Pattern
Every UI Pattern has a corresponding Test Pattern
Both follow the same formal definition presented on the last slide

16. The iMPAcT Tool for Android Testing
Same formal definition, but different meanings
UI Pattern
P – when to verify pattern exists
A – Actions to execute to verify pattern is present
C – validates the presence of this UI Pattern
Test Pattern
P – when to test (determined by checking if corresponding UI Pattern exists)
A – Actions to execute if corresponding UI Pattern is correctly used
C – Final assertion if the test passes

17. The iMPAcT Tool for Android Testing
Example UI Patterns:
Side Navigation Bar:
Goal: “Side bar exists”
V: None (empty set)
A: [Read the screen]
C: {“side bar exists but is hidden”}
P: {true} (side bar should always exist)

18. The iMPAcT Tool for Android Testing
Example Test Patterns:
Side Navigation Bar:
Goal: “Side bar takes up full height of screen”
V: None (empty set)
A: [Read screen, open side bar, reread screen]
C: {“side bar takes up full height of screen”}
P: {UI Pattern present AND side bar available AND Test Pattern has not been applied yet}

19. The iMPAcT Tool for Android Testing – How it works

20. The iMPAcT Tool for Android Testing
Results shown in Finite State Machine
Easy to visualize how to recreate errors
Shows actions taken that lead to errors
Takes ~20 minutes to run

21. Conclusions
iMPAcT is superior to the Targeted Event Sequence Generation
Newer paper, built off its results
iMPAcT does not require source code of the application
iMPAcT executes faster
iMPAcT provides its results in a more readable fashion
iMPAcT determines failures that aren’t just app crashes
Targeted Event Sequence Generation can only determine if the app can reach the event or not (only detects crashes)
iMPAcT can have richer tests

22. Conclusions
iMPAct has good results and use cases, but it could be improved
Needs to be extended to IOS and Windows applications
Could also be extended to desktop / web applications
Only tests User Interface Patterns
Would be interesting to apply it to other cases to have an all in one test suite:
Security
Database structure
Resources used (power consumption, gyroscope, accelerometer, etc)

23. Any Questions?

24. References
Main Paper: I. C. Morgado and A. C. R. Paiva, "The iMPAcT Tool: Testing UI Patterns on Mobile Applications,"  th IEEE/ACM International Conference on Automated Software Engineering (ASE), Lincoln, NE, 2015, pp doi: /ASE URL:  number=
Supporting Paper: Casper S. Jensen, Mukul R. Prasad, and Anders Møller Automated testing with targeted event sequence generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis (ISSTA 2013). ACM, New York, NY, USA, DOI: