Software Engineering Research paper presentation Ali Ahmad Formal Approaches to Software Testing Hierarchal GUI Test Case Generation Using Automated Planning

Formal Approaches to Software Testing P. Dasiewicz IEEE Canadian Conference on Electrical and Computer Engineering 2002

Introduction Software Testing: checking and validating the correctness of software Time consuming Difficult Error prune Formal methods Specifying and verifying software systems using mathematical and logical approaches Target: proof of correctness Some verification techniques Model Checking State space exploration

Introduction (cont’d) Current methods are not feasible Too much manual effort Models that are very complex for analysis It is needed to overcome the following The state explosion problem Cost of creating models Applicability: Design level Less complexity May reduce maintenance costs

Introduction (cont’d) UML Simple, general purpose, visual modeling Specify, visualize, construct and document UML-statechart  basis for dynamic behavioral description Objectives: Overview model checking to the validation and test case generation (using UML statecharts and interaction diagrams) Applicability of formal methods to component integration and interaction testing

Integration/Component Level Testing Offutt Changes in component states due to change events UMLTEST, integrated with Rational Rose  Highly effective test cases can be generated for system level specification Test prefix: ensure that the system a certain pre state before performing the test Issues: Interaction is not considered Some states may never been entered Incorporated UML collaboration diagrams

Integration/Component Level Testing (cont’d) Yoon Test case generation based on UML sequence and collaboration diagrams A node that represents both integration target and message flow Testing technique Extract sequence diagram If concurrency, extract collaboration diagram Divide into ASF (Atomic System Function) Extract nodes and message flows Obtain test cases by applying test criteria (e.g. all-edge) Limitations May require a large number of sequence diagrams Test case coverage (only normal or abnormal flow)

Integration/Component Level Testing (cont’d) Kim Divide statecharts into extended finite state machines Control flows are represented as paths Coverage criteria Path coverage State coverage Transition coverage Test cases breadth or depth first searches can be generated using the data flow on the UML statechart Determine whether class implements correct control and data flow

System/Integration Testing Hartmann Testing the interfaces among several components UML statecharts  Mealy Finite State Machines with restricted point to point synchronous communication model Incremental composition and reduction algorithm Test cases: category partition method Limitations Interaction are modeled as synchronous communications Event exchanges contain no parameters No support for nested machines

Source Code Based Testing Major problem: State space explosion problem Reducing the state space entails eliminating irrelevant code segments Hatcliff Program slicing techniques Used by Bandera for translating Java code to Jimple, e.g. SPIN, SMV  Generates a finite state model for the reduced code Java Path Finder Integrates model checking, program analysis and Testing Goal: Apply formal methods at source level Initially, check for safety properties like deadlocks (LockTree)

Translation of Statecharts Promela: input modeling language for SPIN C-like, extended with non-deterministic, and loop guarded constructs Latella presents a translation from UML statecharts into Promela, where statecharts are first converted to hierarchal automata. A proof of correctness is also given

Component Interaction Testing Major issue: Are the components developed separately work properly together Formal methods of component interaction Define testing requirements Automatically generate the test cases using model checking Focus of author’s research: Component interaction of software systems Detect subtle interaction errors without duplicating the unit level Underlying model: Labeled transition system ObjectState: modeling language to show the feasibility of creating formal models, it relies on hierarchal FSM.

Hierarchal GUI Test Case Generation Using Automated Planning Memon A., Pollack M., Lou Soffa M. IEEE Transactions on Software Engineering 2000

Introduction GUI testing is difficult The interaction space is enormous Determining the coverage of test cases Regression testing is a major challenge Automation is necessary for generating GUI test cases PATHS: Planning Assisted Tester for grapHical user interface Systems Input: Possible goals for GUI user Generates sequences of events to satisfy the user goals, these become test cases

Introduction (cont’d) PATHS performs automated analysis of the hierarchal structure of the GUI to create hierarchal operators that will be used during plan generation Contribution to Research Make use of an AI technique (Automated Planning) Exploits GUI structural features Makes Regression testing easier The test suite is portable Allow the reuse of operator definitions that commonly appear across GUIs

Overview GUI consists of components like labels, buttons, menus, and pop-up lists Example: Microsoft WordPad GUI has 2 types of windows GUI windows Object windows, doesn’t contain any window components

Overview (cont’d) PATHS- Plan Generation Inputs Initial state Goal state A set of operators applied to a set of objects Preconditions and effects Solution to a Planning Problem: sequence of instantiated operators which result in the goal state when executed in the initial state 2 Phases Setup Create a hierarchal model of the GUI Plan Generation Specify scenarios (initial and goal states) PATHS generate a test suit for the scenarios

Overview (cont’d) Role of Test Designer and PATHS during test generation GUI Events and Planning Operators

Overview (cont’d) Operator Event Mapping Example operator: Edit_Cut Abstract: Invokes a window that monopolizes the user interaction

Overview (cont’d) High level plan Expanded plan

Plan Generation GUI test case generation can be modeled by hierarchal plans that doesn’t require conflict resolution

Planning GUI Test Cases Developing a representation for the GUI and its operators Operator Derivation Process Traverse the GUI and press on its components, the label is read off the component label GUI events Menu open (File, Edit) Unrestricted focus (Basic shapes in PowerPoint) Restricted focus (Edit-Preferences in PowerPoint) System-interaction (Cut and paste) Planning Operators System-Interaction Abstract Modeling Initial and Goal States and Generating Test Cases using algorithm Generating multiple plans in PATHS Creating multiple linearizations of the partial plans Repeating the planning process, thus generating a different test case

Planning GUI Test Cases (Cont’d)

Some Experimental Results Multiple Tasks Hierarchal vs. Single Layer

Questions