1. SimplyDroid: Efficient Event Sequence Simplification for Android Application
Speaker: Bo Jiang*
Co-authors: Yuxuan Wu*, Teng Li*, W.K. Chan†
*Beihang University
†City University of Hong Kong
Homepage of Bo Jiang:

2. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

3. Bugs in Apps v.s. Testing Techniques
Fuzz Testing
GUI Traversal-based Testing
Search-based Testing …

4. Fuzz Testing and the Problem
Monkey Tools and its Variants
Failure Triggering Event Sequence is too long !
Hard to replay and analye in debugging
monkey is good in that it can automatically generate and test without human effort
however, analyze test report and debug is hard

5. Fuzz Testing & its Problem
Monkey Tools and its Variants
Failure Triggering Event Sequence is too long !
Hard to replay and analyze…
How to simplify the event sequence while triggering the same failure?

6. Contents Background Motivation Design and the Algorithm of SimplyDroid
Experiments and Results Analysis
Related work
Conclusion & future work

7. Test Case Reduction can Help
Dalvik Explorer
ActivityNotFoundException
Before reduction: 1525 GUI input events
(one launch app, 1524 input events)
After reduction: 4 GUI input events
(one launch app event, 3 touch inputs) 7

8. Test Case Reduction can Help
Dalvik Explorer
ActivityNotFoundException
Before reduction: 1525 GUI input events
(one launch app, 1524 input events)
After reduction: 4 GUI input events
(one launch app event, 3 touch inputs) 8

9. Test Case Reduction can Help
Dalvik Explorer
ActivityNotFoundException
Before reduction: 1525 GUI input events
(one launch app, 1524 input events)
After reduction: 4 GUI input events
(one launch app event, 3 touch inputs) 9

10. Delta Debugging and its Problem
Delta Debugging (DD) by Andreas Zeller : partitions an event sequence into subsequences with equal size, and performs trial execution for reduction with increasingly finer granularity.
DD’s View: a flat structure …… 10

11. Delta Debugging and its Problem
Delta Debugging (DD) by Andreas Zeller : partitions an event sequence into subsequences with equal size, and performs trial execution for reduction with increasingly finer granularity.
DD’s View: a flat structure ……
DD cannot make use the structure of the input for efficiency 11

12. The Hierarchical Delta Debugging
HDD: view the input as a hierarchical tree structure, reduce subtrees from top down (Misherghi and ICSE 2006)
C program for compiler, xml/html for web servers 12

13. The Hierarchical Delta Debugging for GUI
Start Gmail
view mail list
vew mail
compose mail
select receiver
input subject
input body
select attachment
click add attachment
browse to directory
select file
click send
Delete mail from list 13

14. The Hierarchical Delta Debugging
Input events can be grouped into levels of sessions
Start Gmail
view mail list
vew mail
compose mail
select receiver
input subject
input body
select attachment
click add attachment
browse to directory
select file
click send
Delete mail from list 14

15. The Hierarchical Delta Debugging for Android
How to automatically group events into sessions to create the hierarchy? 15

16. User Interaction Session : Compose Mail1 6 2 3 4 5 16

17. User Interaction Session : Compose Mail1 6 2 3
A heuristic: sequence of events forms a session when their starting GUI state is equivalent to their arriving GUI states. 4 5 17

18. Categorize Input Events1 6 2 3
May record the GUI state before each input event to help group events 4 5 18

19. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

20. Overall Workflow of SimplyDroid
Android Device
Testing Host
ADB
Enhanced Monkey
with State Logger
States Log
Test Case Reduction Engine
SimplyDroid
Testing results
Application
Monkey Scripts
Reduced Test Case
Test Case
(Crash Trace)
Results Checker
Android OS
Open Sourced:

21. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

22. Building the GUI State Hierarchy Tree1 22 18 2 7 17 21 …… …… …… 3 6 8
9-15 16 19 20 23 24
25-35 36
Virtual Root Node
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
DalvikExplorerActivity: the entry Activity
BuildActivity: device, system, kernel
DeviceActivity: cpu, memory
Tree Structure
EnvironmentVariablesActivity
State Transition by User Input
TimeZoneActivity
Crash State

23. Building the GUI State Hierarchy Tree1 22 18 2 7 17 21 …… …… …… 3 6 8
9-15 16 19 20 23 24
25-35 36
Each node i represents a GUI state and the event triggered on it.
Virtual Root Node
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
DalvikExplorerActivity: the entry Activity
BuildActivity: device, system, kernel
DeviceActivity: cpu, memory
Tree Structure
EnvironmentVariablesActivity
State Transition by User Input
TimeZoneActivity
Crash State

24. Building the GUI State Hierarchy Tree1
Node 1 represents a virtual initial GUI state
and the 1st event to launch the app.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
The log of the monkey: <s1, e1, s2, e2, s3, e3, s4, e4, s5, e5,s6, e6…scrash>

25. Building the GUI State Hierarchy Tree1 2
Node 2 represents the entry activity and the event triggered on this Activity
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
The log of the monkey: <s1, e1, s2, e2, s3, e3, s4, e4, s5, e5,s6, e6…scrash>

26. Building the GUI State Hierarchy Tree1 2 3
Node 3: the state is compared on the path from toot node to node 2. It is not equivalent to node 1 or node 2, so add it as the child node of node 2.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
The log of the monkey: <s1, e1, s2, e2, s3, e3, s4, e4, s5, e5,s6, e6…scrash>

27. Building the GUI State Hierarchy Tree1 2 3 4
Node 4: the state s4 is compared on the path from toot node to node 3. It is equivalent to node 3, so add it as the sibling node of node 3.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.
The log of the monkey: <s1, e1, s2, e2, s3, e3, s4, e4, s5, e5,s6, e6…scrash>

28. Building the GUI State Hierarchy Tree1 2 …… 3 6
Node 6: sibling of node 3&4&5.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

29. Building the GUI State Hierarchy Tree1 2 …… 3 6
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

30. Building the GUI State Hierarchy Tree1 2 7 …… 3 6
Node 7: sibling of node 2.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

31. Building the GUI State Hierarchy Tree1 18 2 7 17 …… …… 3 6 8
9-15 16 19 20
The tree always grows at right most path from root
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

32. Building the GUI State Hierarchy Tree1 18 2 7 17 21 …… …… 3 6 8
9-15 16 19 20
The tree always grows at right most path from root
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

33. Building the GUI State Hierarchy Tree1 22 18 2 7 17 21 …… …… …… 3 6 8
9-15 16 19 20 23 24
25-35 36
A partial GUI State Transition Tree for Input Events
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

34. Building the GUI State Hierarchy Tree1
View Device
View Build 22 18 2 7 17 21 …… …… …… 3 6 8
9-15 16 19 20 23 24
25-35 36
View
device model
Back to
entry Activity
View OS inf
Back to entry Activity
With the tree, events are grouped into hierarchy of sessions based on their preceding GUI state.
For each state si and event ei with the same number i, a new node n is constructed. Then node n is compared with each node (state) m from root to its previous node si-1 in turn. If si is equal to any node m, node n is added as a sibling node of m. Otherwise, n is added as a child node of node si-1.

35. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
The Construction of GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

36. The HDD Algorithm (Hierarchical Delta Debugging)1 2 …… …… ……
Reduction at each level from top down, level 1 first

37. The HDD Algorithm 1 2 …… …… ……
Equal partition at level 1

38. The HDD Algorithm 1 2 ……
The second partition can trigger CRASH in trial execution

39. The HDD Algorithm 1 2 ……
continue partition at level 1 on the retained events

40. The HDD Algorithm 1 2 ……
The second partition can trigger crash and is retained
level 1 is done1

41. The HDD Algorithm 1 2
Reduction at level 2, two partitions first

42. The HDD Algorithm 1 2
The last partition cannot trigger CRASH

43. The HDD Algorithm 1 2
Refined partition into 4

44. The HDD Algorithm 1 2
Try different combination until triggering CRASH

45. The HDD Algorithm 1 2
Try different combination until triggering CRASH

46. The HDD Algorithm 1 2
Try different combination until triggering CRASH

47. The HDD Algorithm 1 2
No more refined solution can trigger CRASH, reduction is done.

48. The reduction results:
The HDD Algorithm 1 2 1 22
The reduction results: 23 36

49. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

50. The BHDD Algorithm (B for Balanced)1 2 …… …… ……
The subtrees of level i are often vary in size, makes the partition at level i unbalanced

51. Equalize the nodes at level i while considering subtress
The BHDD Algorithm 1
(5)
(10)
(1)
(3)
(1)
(15)
(19)
(16) 2 …… …… ……
Equalize the nodes at level i while considering subtress
It is the same as HDD in other aspects

52. The BHDD Algorithm 1 2 ……

53. The reduction results:
The BHDD Algorithm 1 2 1 22
The reduction results: 23 36

54. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

55. The LHDD Algorithm (L for Local)1 2 …… …… ……
Reduction from top down, level 1 first
Observation:
Crash node is always the last node at bottom level
Ancestor of the crash node also last node in their level

56. The LHDD Algorithm (L for Local)1 2 …… …… ……
Heuristic of LHDD:
Ensure transition from the ancestor nodes of crash node to the first node of its next level is preserved

57. The LHDD Algorithm (L for Local)1 2 …… …… ……
Checking crash is reduced to checking state transition.

58. The LHDD Algorithm (L for Local)1 ✕ 2 …… …… ……
Optimization 1: Partial Local Reduction.
The large Subtree of last node is not included when reduction at one level.
Checking transition rather than crash

59. Optimization 2 : backward Pre-selection at each level
The LHDD Algorithm 1 2 …… …… ……
Optimization 2 : backward Pre-selection at each level

60. Optimization 2 : backward Pre-selection at each level
The LHDD Algorithm 1 2 …… …… ……
Optimization 2 : backward Pre-selection at each level

61. Optimization 2 : backward Pre-selection at each level
The LHDD Algorithm 1 2 …… …… ……
Optimization 2 : backward Pre-selection at each level

62. Optimization 2 : backward Pre-selection at each level
The LHDD Algorithm 1 2 …… …… ……
Optimization 2 : backward Pre-selection at each level

63. The LHDD Algorithm 1 2 ? …… ……1 2 ? …… ……
When processing level 1, only check whether the reduction can reach the next activity , the events after TimeZoneActivity is not added during reduction

64. The LHDD Algorithm 1 2 ? …… ……1 2 ? …… ……
Preselection, select events from back to front, the last event is chosen first. 1, 2, 4,8,16,…

65. The LHDD Algorithm 1 2 √
After trial execution, finding the last node can reach the next Activity.

66. The LHDD Algorithm 1 √ 2 ……
Add the events after the last Activity, check whether crash can be triggered. Preprocessing is done.
Since there is only one node after preprocessing, no more reduction is needed in this level. The first level is done.

67. The LHDD Algorithm 1 2
Start processing level 2. First, preprocessing, select node from back to front. The last node cannot trigger crash.

68. The last 2 node still cannot trigger crash
The LHDD Algorithm 1 2
The last 2 node still cannot trigger crash

69. The LHDD Algorithm 1 2
The last 16 node exceeds the number of all nodes in this layer. All nodes in the current level is pre-selected. Start reduction on this level.

70. The LHDD Algorithm 1 2 Start reduction, 2 partitions1 2
Start reduction, 2 partitions
The follow-up in level reduction is same as HDD

71. The LHDD Algorithm 1 2
No more reduction can trigger CRASH. Reduction is done.

72. The reduction results:
The LHDD Algorithm 1 2 1 22
The reduction results: 23 36

73. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & future work

74. Research Questions
RQ1: Are HDD, BHDD, and LHDD effective to reduce the size of Android input event sequence?
RQ2: Are HDD, BHDD, and LHDD efficient when performing test case reduction?
RQ3: If LHDD is efficient, which of the two optimizations contributes more to its performance improvement.

75. Subject Programs & Crash Traces
Subject Description
Program Version
Android Version
Fault Type
Device
Exception Type
Number of crash traces
Crash Trace Length
Yahtzee
Game
1.1
2.3.3
Mutant
Mi5
ArithmeticException 17
K9mail
Mail 5
NullPointerException 23
DalvikExplorer
System information Viewer
3.4
4.1.2
Real
Mi1
ActivityNotFound &
OutOfMemoryError 11
WeightChart
Weight recorder
1.0.4
ActivityNotFound 8
16-765
Ringdroid
Ringtone editor
2.6
RuntimeException 6
Tippy
Calculator
1.1.3 9
31-162
SyncMyPic
Photo synchronizer
0.15
27-462
WhoHasMyStuff
Item lending helper
1.0.7
4.4.2 10
50-679

76. Subject Programs & Crash Traces
Subject Description
Program Version
Android Version
Fault Type
Device
Exception Type
Number of crash traces
Crash Trace Length
Yahtzee
Game
1.1
2.3.3
Mutant
Mi5
ArithmeticException 17
K9mail
Mail 5
NullPointerException 23
DalvikExplorer
System information Viewer
3.4
4.1.2
Real
Mi1
ActivityNotFound &
OutOfMemoryError 11
WeightChart
Weight recorder
1.0.4
ActivityNotFound 8
16-765
Ringdroid
Ringtone editor
2.6
RuntimeException 6
Tippy
Calculator
1.1.3 9
31-162
SyncMyPic
Photo synchronizer
0.15
27-462
WhoHasMyStuff
Item lending helper
1.0.7
4.4.2 10
50-679
8 subject programs, 92 deterministic crash traces
Our algorithm only worked on deterministic traces, so we examined these 120 crash traces to check whether they can replay stably. We manually executed each crash trace 3 times
Finally, the 92 crash traces left were all used in our data analysis.

77. Experiment Setup Hardware Procedure for RQ1 and RQ2 Procedure for RQ3
Lenovo V4400, Intel i and 16GB memory
Procedure for RQ1 and RQ2
Reduce 92 traces with HDD, BHDD, LHDD
Record the size of reduced test cases (RQ1)
Record the reduction time (RQ2)
Procedure for RQ3
Reduce 92 traces with LHDD-NoPre
i.e., Turn off pre-selection
Record the size of reduced test cases
Record the reduction time

78. Research Questions
RQ1: Are HDD, BHDD, and LHDD effective to reduce the size of Android input event sequence?
RQ2: Are HDD, BHDD, and LHDD efficient when performing test case reduction?
RQ3: If LHDD is efficient, which of the two optimizations contributes more to its performance improvement.

79. Answering RQ1: Reduction Effectiveness

80. Answering RQ1: Reduction Effectiveness
HDD, BHDD, and LHDD are comparable to DD in Reduction Effectiveness.

81. Research Questions
RQ1: Are HDD, BHDD, and LHDD effective to reduce the size of Android input event sequence?
RQ2: Are HDD, BHDD, and LHDD efficient when performing test case reduction?
RQ3: If LHDD is efficient, which of the two optimizations contributes more to its performance improvement.

82. Answering RQ2: Reduction Efficiency
>  DD
HDD
BHDD
Count
Percentage 55
60% \ 75
81% 71
77%
LHDD 86
93% 84
91% 74
80%
we have computed the number and percentage of crash traces where the former uses less test case reduction time (in other words, better “>”) than the later

83. Answering RQ2: Reduction Efficiency
>  DD
HDD
BHDD
Count
Percentage 55
60% \ 75
81% 71
77%
LHDD 86
93% 84
91% 74
80%
Reduction Efficiency: LHDD > BHDD > HDD > DD
“>” = more efficient

84. Research Questions
RQ1: Are HDD, BHDD, and LHDD effective to reduce the size of Android input event sequence?
RQ2: Are HDD, BHDD, and LHDD efficient when performing test case reduction?
RQ3: If LHDD is efficient, which of the two optimizations contributes more to its performance improvement.

86. Similarly, the crash trace No. 5 and No
Similarly, the crash trace No. 5 and No. 16 of K9mail popped ups floating windows during execution,

89. For example, the crash trace No. 1 and No
For example, the crash trace No. 1 and No. 6 of Ringdroid optionally popped up an AlertDialog, which hided the underlying Activity.

94. Threat to Validity
We selected only stable crash traces to perform test case reduction.
The definition of state equivalence: Activity ID.
Connecters of our tool.

95. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & Future Work

96. Related Work Delta Debugging Berkeley Delta C-Reducer
ACM SIGSOFT Software Engineering Notes, TSE 2002.
Equal partition and reduction
String input & character level reduction
Berkeley Delta
Daniel S. Wilkerson, Scott McPeak, Alex Aiken and George Necula
Reduction input for C++ front end
Line level reduction
C-Reducer
John Regehr et PLDI 2012
Modularized reduction on C grammar tree.

97. Related Work Hierarchical Delta Debugging
Misherghi and ICSE 2006
Efficient Reduction for hierarchical structure
XML/HTML file & /C/C++ programs
Non-Deterministic Delta Debugging Minimization
Clapp et FSE 2016.
Minimize event trace to reach a given target Activity
Address non-deterministic execution
Probability approach: execute candidate reduction many times

98. Contents Background Motivation
Design and the Algorithms of SimplyDroid
The Overall Design
Grouping Events with GUI State Hierarchy Tree
The HDD Algorithm
The BHDD Algorithm
The LHDD Algorithm
Experiments and Results Analysis
Related Work
Conclusion & Future work

99. Conclusion
In this work, we have proposed SimplyDroid, a crash trace simplification tool for Android applications.
We have proposed a novel GUI state hierarchy tree to group events into sessions automatically.
We proposed a family of 3 hierarchical delta debugging algorithms to operate on this trace representation.
Our experiments show that techniques in this family are increasingly more efficient to perform test input reduction.

100. Future Work
For future work, we will further study the non- deterministic execution problem in the context test case reduction.
We will also perform empirical study to explore generality of technique for GUI application.

101. Thank you!

102. Thank you!

103. Thank you!