1. Software Testing for Non-traditional Software
Mary Lou Soffa
University of Pittsburgh
Currently on sabbatical at
IBM T.J. Watson Research
Shanghai, China
December 4, 2003

2. Research Interests Software Engineering Optimizing compilers
Testing
Regression testing - maintenance
Debugging
Optimizing compilers
Particular code optimizations
Properties of optimizations
Specification/profitability
Connection: program analysis

3. Non Traditional Software
Input
Program executes without user interaction and is self contained
Program produces results
Non Traditional
Event driven – user interacts with code as it executes – Graphical User Interfaces
Application operates in an environment
Database, physical environment

4. Testing Graphical User Interfaces
Outline
Testing Graphical User Interfaces
Challenges – how currently done
Test case generation
Oracles – expected results
Regression testing

5. Testing of Graphical Unit Interfaces (GUI)
50% of code
GUI
Interactions between the
GUI and the underlying code
Underlying
Code

6. Challenges of GUI Testing
How much testing is enough?
(Test coverage)
Is the GUI executing correctly during testing?
(Test oracles)
What test results can be salvaged from the previous test runs to test a new version?
(Regression testing)
How to represent the GUI to handle all the above?
(Representation)
How to do the testing automatically without user’s involvement?

7. GUI Framework New representation for GUIs – event driven software
Test case generation based on AI planning
Unique test oracles
New event-based coverage criteria
New automated regression testing algorithms
Automated test executor
A comprehensive framework that integrates all the above components

8. Overview of the Framework
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Representation
GUI
Implementer
Executing
GUI
Regression
Tester
Test
Oracle
Specifications may be formal or informal
Representation used by all techniques
Test Case
Generator
Test
Coverage
Evaluator
Test
Executor

9. Tools (Languages/Toolkits)
GUI Representation
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Implementer
Test Designer
GUI Model
Executing
GUI
Regression
Tester
Test
Oracle
The GUI model is used by all the components of the framework.
I will focus on those aspects of the model that are needed for the test case generator and test oracles.
Test Case
Generator
Test
Coverage
Evaluator
Test
Executor

10. Creating the GUI Model Modeling
GUI’s state in terms of objects & their properties
Events as state transducers
GUI’s hierarchical structure
The test designer creates a model of the GUI. The model should capture necessary aspects of the GUI so as to ensure efficient test case generation and test oracles.
It is important to create a model of the state of the GUI so that we know exactly what the GUI does.
Events are actions that the user performs on the GUI. These events change the state of the GUI. We model them as state transducers.
Experience with conventional software has shown us that a hierarchical representation allows decomposition of problems and their solutions. In testing conventional software, procedures/methods are unit tested. Then integration testing is performed. GUIs are no different. They are developed in a hierarchical manner. Components are used from class libraries and incorporated into the GUI leading to a hierarchical structure.
I will show how a classification of the events in the GUI leads to a hierarchical decomposition from the test designer’s point of view.

11. Modeling the GUI’s State
A GUI at time T is modeled using:
Objects O = {o1, o2, o3, …, om}
Properties P = {p1, p2, p3, …, pl}, where pi is an ni-ary (ni >= 1) Boolean relation of the form:
Property(oj, oa, ob, …,ox, value)
True/False
Optional Objects
Caption(Button, “Cancel”)
Property Name
Object

12. Example: Modeling the GUI’s State
Form1
WState(Form1,
Width(Form1,
Scroll(Form1,
wsNormal)
1088)
TRUE)
Button1
Label1
Caption(Button1,
Enabled(Button1,
Visible(Button1,
Height(Button1,
Cancel)
TRUE)
65)
Align(Label1,
Caption(Label1,
Color(Label1,
Font(Label1,
alNone)
“Files of type:”)
clBtnFace)
(tfont))

13. GUI Events GUI’s state is not static Events change the GUI’s state
Events E = {e1, e2, e3, …, en}, associated with a GUI are functions from one GUI state Si to another state Sj
Notation: Sj = e1(Si)

14. Example: An Event State: Si Event: e State: Sj Sj = e(Si) SelectText
This is the text.
State: Si
This is the text.
This
SelectText
(“This”)
Event: e
State: Sj
Sj = e(Si)

15. Now we know how to represent the events of the GUI
Operator Example
Operator :: CUT
Precondition:
isCurrent(Menu2).
Effects:
FORALL Obj in Objects
Selected(Obj) 
ADD inClipboard(Obj)
DEL onScreen(Obj)
DEL Selected(Obj)
ADD isCurrent(Menu1)
DEL isCurrent(Menu2).
Menu1
Cut
Menu2
Primitive Operators
Now we know how to represent the events of the GUI

16. GUI Modeling Steps From the GUI’s specifications (formal/informal),
Identify objects and properties
Create operators for GUI events
Using the event classification, create hierarchical and system-interaction operators
The model is used to develop all techniques to test GUIs.
Now I describe the use of the model for test case generation.

17. Tools (Languages/Toolkits)
Test Case Generator
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Implementer
Test Designer
GUI Model
Executing
GUI
Regression
Tester
Test
Oracle
Tasks
Test-Case
Generator
Test
Coverage
Evaluator
Test
Executor
Test Case

18. GUI Test Cases Individual user events
Not enough !
Sequences of user events lead to different states
Test case: sequence of user events
What is a GUI test case?
Simplest approach is to use individual events.

19. A Test Case for WordPad This is the text. S0 This is the text.
SelectText
(“This”)
Format
Font 18 OK
(“text”)
Underline
This is the text.

20. Selecting Test Sequences
Infinitely many
Randomly choose sequences
Expert chooses sequences
Automatically generate events for COMMONLY USED TASKS
This is exactly what an AI planner does.
The approach in the work is to use the planning techniques to generate test cases.
Multiple
Event
Sequences
This is the text.
This is the text.
Initial State
Goal State

21. A Plan for a GUI Task This is the text. SelectText(“This”)
SetFontSize(18)
is the
This
text.
Initial State
Goal State
This is the text.
This is the text.
SelectText(“text”)
MouseClick(U)
is the
This
text.

22. Overview of Test Generation

23. Using Primitive Operators
One operator for each event
Operator :: CUT
Precondition:
isCurrent(Menu2).
Effects:
FORALL Obj in Objects
Selected(Obj) 
ADD inClipboard(Obj)
DEL onScreen(Obj)
DEL Selected(Obj)
ADD isCurrent(Menu1)
DEL isCurrent(Menu2).
File Edit View Ins
Menu1
Cut
Menu2

24. Create System-Interaction Operators
File
Send To
Sys-Interaction Operator:
File_SendTo_MailRecipient
= <File + SendTo + MailRecipient >
Mail Recipient

25. Effects of Exploiting the GUI’s Structure
Reduction in planning operators
362 operators  32 operators
Ratio 10:1 for MS Wordpad
20:1 for MS Word
Efficiency
The ratio for MS Word is much higher. It has more structure and depth.

26. Initial
State
This is the text.
Goal
State
This is the text.

27. Test Case INITIAL Primitive Operator Hierarchical Operator Primitive
GOAL
SelectText
(“This”)
FormatFont
(“This”, 18pt)
SelectText
(“text”)
FormatFont
(“text”, Underline)
FormatFont 18 OK
Planner
FormatFont
Underline OK
Planner
Format
Font
Mapping
SelectText
(“This”)
Format
Font 18 OK
(“text”)
Underline

28. Test Case Generation

29. Automated Execution
Human effort: define operators: 2 days. Define tasks: one day.
The human effort and practical performance shows that the technique is practical and effective.

30. Tools (Languages/Toolkits)
Test Oracles
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Implementer
Test Designer
GUI Model
Executing
GUI
Oracle
GUI State
(run-time)
Regression
Tester
Expected-state
Generator
Tasks
The same GUI model is also used in developing test oracles.
Expected-state
sequence
Test-Case
Generator
Execution
Monitor
Test
Coverage
Evaluator
Verifier
Test Case
Actual State
Information
Verdict
Test Executor

31. What Is Correct Behavior
This is the text.
This is the text. ?!
SelectText
(“This”)
Format
Font 18 OK
(“text”)
Underline
This is the text.
Check State, not only Output !!

32. Determine Correct Behavior
To check the GUI’s state after each event
Approaches
Manual
Automated
Actual State
Expected State
Automated
Verifier
Verdict
Challenges
Generating expected state
Extracting actual state
Comparing expected & actual states

33. Overview of GUI Oracle Oracle Expected-state Generator Execution
Test Case
Run-time
information from
executing GUI
Expected-state
Generator
GUI
Model
Expected State
Execution
Monitor
Actual
State
Verifier
Oracle
Verdict

34. Expected State Obtaining Next State e1 e2 e3 en
Given a test case T with S0, the initial state, and
A sequence of events
The next state Sy is obtained from the current state Sx and operator Op
Delete property P from Sx if Effects(Op) contains the command “DEL P”
Add property P in Sy if Effects(Op) contains the command “ADD P” e1 e2 e3 en
In order to derive the expected state

35. Deriving Expected State
Obtain S1 = e1(S0)
And Si = ei(Si-1) e1 e2 e3 en S0 S1 S2 S3 Sn
Expected state is a conjunction of properties of the form
Property(Objects…, Value)

36. Experiments Purpose: determine Experimental design
Time to derive expected state
Time to execute monitor and verifier
Experimental design
GUI: our version of MS Wordpad (36 modal windows, 362 events)
Test cases: generated 290 test cases (6-56 events) using an AI planner
Hardware platform: 350 mhz pentium based machine, 256 MB RAM
Properties: reduced set
Level of checking: GUI relevant property set

37. Deriving Expected State
Total CPU time (all 290 test cases and expected states)
75.84 sec.

38. GUI Relevant-properties checking Total running time < 10 minutes
Automated Execution
Without user intervention
GUI Relevant-properties checking
Total running time < 10 minutes

39. Tools (Languages/Toolkits)
Regression Testing
GUI Specifications
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Implementer
Test Designer
GUI Model
Executing
GUI
Oracle
GUI State
(run-time)
Expected-state
Generator
Regression
Testing
Algorithms
Tasks
Summarize: I have presented the overall framework for testing GUIs.
Contributions: The GUI model, test case generator, test oracle, test executor, regression tester, coverage evaluator
Expected-state
sequence
Test-Case
Generator
Coverage
Criteria
Coverage
Evaluation
Algorithms
Execution
Monitor
Verifier
Test Case
Coverage
Report
Actual State
Information
Verdict
Test Executor

40. GUI Regression Testing Problem
Adobe Acrobat 5.0
File
Document Security
A test case of length 3 OK
I will start by given this example that demonstrates a regression test for GUI.

41. GUI Regression Testing Problem
Adobe Acrobat 6.0
File
Document Security
A test case of length 3 OK
So have a choice – can either throw away the test case and generate a new one or repair the test case.
Event “Document Security”
no longer in menu
Test case cannot be executed – unusable!!!

42. A New Test Case Repair the old test case to obtain this new one File
Security
Document Properties .
Repair the old test case
to obtain this new one OK

43. Overview of our Regression testing
Traditional regression testing
Event-flow/component model
Repairing test cases
Case studies
For the rest of the talk, I´ll give a very brief overview of regression testing for traditional software, a field that received much research attention

44. Traditional Regression Testing
original test suite
usable test cases
unusable (obsolete) test cases
discard
select
Selected usable
test cases
new
test cases
In regression testing of traditional software, after a change is made to software, the test suite is decomposed into two parts
Usable test cases
Unusalbe – obsolete – discarded – because of the change, they do not test what they were designed to test
Usable test cases – retest all or select which ones to test –
Priority
regression test suite
Regression test-all
Selective regression testing process

45. Regression Testing for GUIs
Test case generation is manual – capture/replay tools
Small number of test cases
Each test case is valuable
Changes result in a large number of obsolete test cases
Structural and layout changes
Frequent modifications
Driven by constant user feedback – prototyping
Need frequent testing
GUI testing has a number of chararacterists that challenges regression testing and also make regression testing very important.
Although we have developed a method for generating test cases, most of the GUI test cases are generated manually, usually using capture replay tools. ( I am told transitioning to industry takes 10 years)
Manual process with capture tools giving some support.
Consequently small number of test cases
Each test is valuable – took considerable amount of time to generate it
Changes made – affect large number of test cases and these test cases are unusable if they changed the structural and layout changes
Also, another characteristic is that GUIs are developed by prototypting and are driven bz user feedback – need frequent testing

46. Our approach GUI Regression Testing
original test suite
usable test cases
unusable test cases
select
repair
not repairable
Selected usable
test cases
new
test cases
repaired
discard
So in our approach to regression testing, after a change is made to a GUI
regression test suite

47. GUI regression model
Model represent all legal sequences of events and event interactions
Two levels
Component-call-graph – built using the concept of modal window
Forms GUI component
GUI event flow graph – event structure of a component
Helps to isolate changes and repairs

48. Representing a Component
File Edit Help
follows
Definition: Event ex follows ey iff ex can be performed immediately after ey.
Open …
Save
Open …
Save
Cut
Copy
Paste
About …
Contents …
Forms event flow graph
What is a component
Everything that can be performed from the component

49. Component-Call Graph Definition: Component-Call-Graph
Main
FileNew
FileSave
FileOpen
PageSetup
Print
ViewOptions
FormatFont
Properties
Definition: Component-Call-Graph
Nodes are components in the GUI
Main is distinguished component
Edges represent the invokes relation between components, i.e., (Cx, Cy) є {edges} iff Cx invokes Cy.
We model the GUI by its components using a tree of events

50. Event Flow Graph Definition: Event Flow FG Vertices represent events
File
Help
Edit
Open
Save
About
Contents
Cut
Copy
Paste
To File, Edit
and Help
To File, Edit
and Help
Definition: Event Flow FG
Vertices represent events
Edges show the follows relationship
Events first available (shown in red)
Events that invoke other components (dotted lines).
They may invoke other components – dashed circles
This representation enables us to focus on only those parts of the GUI that have been changed.

51. To repair test cases Two phases Checker Repairer
Determine modifications made to original GUI
categorizes test case usable or unusable
Repairer
Determines if unusable cases repairable
If yes, repairs test case
Note: can be done in same step in implementation
Using these models, we can now repair test cases. Our repair has two phases
Check phase that examines the original and modified GUI and determines the changes that have been made
Explores the existing test cases and determines which are usable and which ones are not
The test cases that are not usable are sent to the repairer, which determines if they are repairable and if they are, it repairs them.
Note I am explaining the technique using 2 phases but both phases can be rolled into one phase for implementation.

52. Checker Automatically determines modifications –
Models for original and modified GUIs
Component call graph
Components & calls added and deleted
Event flow graph (per component)
Events & interactions added and deleted
Additions do not cause a test case to become unusuable but deletions do
One GUI modification – multiple changes in test case
Checker considers both the Component call graph and event control flow graph of components and determines the changes.
Classifies these as events and interactions (edges) added and deleted
Does the same for the component call graph/
Additions either of events, components or edges do not make the test case unusable but deletes do

53. Repairer
Takes unusable test case sequence: searches from the deleted node/edge
Events and edges that can be deleted and lead to a legal sequence
An event that can be inserted that leads to a legal sequence with deletion
More than one way to repair – do all to produce more test cases
Repairer – similar to error recovery of a parser
Attempts to repair the test case by
either deleting events and edges until it finds a legal sequence
Adding events and edges – restrict to one event to keep the test case similar and the efficiency reasonable.

54. Repairing Process A deleted event – delete to repair e1 ... ek ei-1 ei
ei+l
ei+m
... en
follows

55. Repairing Process A deleted event – insert to repair e1 ... ek ei-1 eiex
ei+1
...
ei+m
... en
follows

56. Repairing process Delete edge ... ei ei+1 ... en e1 ei-1 ... followsek en
delete
... e1
...
ei-1 ei
add e1
...
ei-1 ei ex en
ei+1
...

57. Case Studies Questions
How many test cases are made unusable by GUI modifications across versions?
How many unusable test cases are repairable?
How much time does the checker and repairing processes take?
We performed some case studies to determine the effectiveness and efficiency of our technique.
Wanted to answer

58. Two Applications Each with two versions Adobe’s Acrobat Reader
version 4.0 (for Linux)
15 components with 176 events (not counting short-cuts)
version 5.0 (for MS Windows 2000)
25 components with 351 events
MS WordPad – our own version
36 modal windows, and 362 events (not counting short-cuts)
Menu added to version 2

59. Adobe application Old and new versions
400 test cases generated manually using a capture/replay tool
Generate test cases: 8 hours
Old and new versions
Checker time: 7 sec
Unusable: 296 (74%) test cases
Repaired: 211 (71%)
Repairer: 18 sec.
Regression test suite: 315 (79%)

60. Wordpad application Old and new version Comparable results for Wordpad
400 test cases
Generate test cases: 5 hours
Old and new version
Checker time: 6 sec
Unusable: 210 (52%)
Repaired: 210 (100%)
Repairer: 18 sec
Regression test suite: 100%

61. Putting it All Together
GUI Specifications
GUI Specifications
GUI Implementation:
Tools (Languages/Toolkits)
GUI
Implementer
Test Designer
GUI Model
Executing
GUI
Oracle
GUI State
(run-time)
Expected-state
Generator
Regression
Testing
Algorithms
Tasks
Summarize: I have presented the overall framework for testing GUIs.
Contributions: The GUI model, test case generator, test oracle, test executor, regression tester, coverage evaluator
Expected-state
sequence
Test-Case
Generator
Coverage
Criteria
Coverage
Evaluation
Algorithms
Execution
Monitor
Verifier
Test Case
Coverage
Report
Actual State
Information
Verdict
Test Executor

62. Conclusions & Future Work
Broaden the testing of GUIs to the testing of event driven systems
Debugging of event driven software