Comparing model-based and dynamic event-extraction based GUI testing techniques : An empirical study Gigon Bae, Gregg Rothermel, Doo-Hwan Bae The Journal of Systems and Software (Volume 97), November 2014, Pages Presented by Xin-Hung Chen,

Outline Introduction Preliminaries GUI testing framework Test case generation Empirical study Conclusion 2

Introduction Graphical user interfaces are pervasive. One obvious way to test GUIs is with “manual” testing. ▫ Time wasting ▫ High cost Researchers have attempted to create GUI testing techniques that automate the processes of generating and executing test cases. 3

Automated GUI testing: ▫ Model-based GUI testing  More effective at detecting faults  Less cost ▫ Dynamic event-extraction based GUI testing  More effective at covering statements. 4 Introduction (cont.)

To facilitate this comparison ▫ A GUI testing framework The overall contributions of this work include the following: ▫ A comprehensive experiment comparing the two kinds of GUI testing techniques. ▫ A flexible GUI testing framework that lessens the potential of implementations. ▫ A better understanding of the relative strengths and weaknesses of the two types of GUI testing techniques. ▫ A discussion and better understanding of the various factors affecting the application of GUI testing techniques. ▫ Guidelines for software engineers and researchers on things to consider. 5

Preliminaries GUI testing ▫ A GUI is composed of a set W of widgets that a user interacts with through a set A of actions. Each action is accompanied by a set V of values. ▫ An event e can be represented by a tuple (w, a, v)  w ∈ W, a ∈ A, and v ∈ V. 6

Preliminaries (cont.) GUI test cases ▫ A test case t is composed of a sequence of events (e 1, e 2,..., e n ). The length of t can be defined as the number of events in t. ▫ t = (click button 5, click button +, click button 2, click button =) 7

Preliminaries (cont.) Event coverage test adequacy ▫ Event-t-tuple:  Definition: An event-t-tuple (e i, e j,..., e t ) is an ordered tuple of size t of events from E. A set of events E gives rise to |E| t event-t-tuples. ▫ t-sequence-cove:  Definition: A test suite is t-sequence-cover adequate if it executes all feasible event-t-tuples by way of an event-t- sequence at least once. 8 event-2-tuples

GUI testing framework 9

GUI testing framework (cont.) 10 Strategy ▫ Example of google map’s path search

Test case generation Static model-based GUI testing ▫ Model creation ▫ Test suite generation ▫ Test suite execution Dynamic event-extraction based GUI testing ▫ Create test case and executes it at the same time 11

Test case generation (cont.) Static model-based GUI testing ▫ Model creation  Algorithm 1  Input: an AUT, a terminal event list  Output: an event flow graph(EFG model) and a set of initial events I 12

Test case generation (cont.) Static model-based GUI testing ▫ Test suite generation  Algorithm 2  Input: an EFG model, a set of initial events I and a test strength T  Output: a test suite TS 13

Test case generation (cont.) Static model-based GUI testing ▫ Test suite execution  Algorithm 3  Input: an AUT and a test suite  Output: a failure report 14

Test case generation (cont.) Dynamic event-extraction based GUI testing ▫ Create and execute test cases  Algorithm 4  Input: an AUT, a test case length K, and a test suite size S  Output: a failure report and a test suite 15

Empirical study To compare techniques, traditional measures such as fault-detection and elapsed time are necessary but not sufficient. ▫ How to compare the cost? 16

Empirical study (cont.) Dependent variables 1.Effectiveness in terms of code coverage 2.Effectiveness in terms of fault detection 3.Cost in terms of numbers of executed events 4.Cost in terms of elapsed time 5.Event sequence coverage 6.GUI State coverage 17

Empirical study (cont.) Experiment procedure 18

Empirical study (cont.) Experiment procedure 19

Empirical study (cont.) Comment 20

21

22

Conclusion Model-based GUI testing ▫ Faults detection ▫ Less elapsed time Dynamic event-extraction GUI testing ▫ State coverage ▫ Not effective for short test cases 23

Conclusion (cont.) The overall contributions of this work include the following: ▫ A comprehensive experiment comparing the two kinds of GUI testing techniques. ▫ A flexible GUI testing framework that lessens the potential of implementations. ▫ A better understanding of the relative strengths and weaknesses of the two types of GUI testing techniques. ▫ A discussion and better understanding of the various factors affecting the application of GUI testing techniques. ▫ Guidelines for software engineers and researchers on things to consider. 24