1. Lab 8: GUI testing
Software Testing LTAT

2. Install & Introduction
This lab requires virtual machine
VirtualBox is preferred
Import the .ova file to your VM
This file contains Windows 10 with Testar
In the Desktop, you find Testar folder, which contains Testar and SUTs for the lab and homework task
Make sure you have enough free space (>= 20 GB)

3. Capture & Replay and Visual testing tools
Selenium, Sikuli
Simple and easy to use
Maintenance problem if user interface changes

4. ‘Scriptless’ testing tool Testar
Scripts are generated automatically
No maintenance issues with regards to scripts
Main purpose is robustness testing
Robustness is the ability of a computer system to cope with errors during execution and cope with erroneous input.

5. Idea of Testar tool (1) GUI contains graphical objects called widgets
Widgets form a hierarchy called widget tree
All widgets have properties (title, etc.)
Widget tree and properies of each widget form a GUI state
GUI State changes, if user executes some action (click, write, etc)
Oracles can be defined to check whether a state has been reached that is suspicious to be incorrect

6. Idea of Testar tool (2) Check oracles Sequence leading to failure
Failure detected
No failure detected
Start SUT
Scan GUI state
Check stop criteria
Stop SUT
Yes No
Execute action
Derive (+filter) actions
Select action

7. Idea of Testar tool (3)
Some parameters have to be set before running Testar (-> stop criteria)
Check oracles
Sequence leading to failure
Failure detected
No failure detected
Start SUT
Scan GUI state
Check stop criteria
Stop SUT
Yes No
Regular expressions are used to configure oracles and filters
Execute action
Derive (+filter) actions
Select action

8. Idea of Testar tool (4) – Stop criteria
Number of Sequences (=Iterations)
Number of Actions per Sequence
Identify when there is enough actions done for one test

9. Idea of Testar tool (5) - Oracles
Testar cannot determine if the SUT functions correctly
For example, it cannot determine that 2+2 does not equal to 5
Testar can detect failures such as crashes and hangings (without oracles)
In addition, Testar can detect failures if widget titles matches oracles
Example: user specifies „fail“ as an oracle
If word „fail“ appears in some state, then Testar reports it as a failure (test = fail)
If word “fail” doesn’t appear in a sequence, then by default the test passes

10. Idea of Testar tool (6) - Filters
By default, Testar will execute actions on all the widgets it can find
Including buttons on the title bar (close, minimize, maximize, system)
User can define that specific widgets cannot be executed
For example, forbid clicking on ‘close’ button
To do this, an action filter can be configured
Filtering is done by matching the widget title to filters
For example, user can specify „.*close.*“ as a filter and Testar will not execute actions on widgets, which title contains „close“

11. Testar cannot determine
Testar can determine

12. Introduction to folder structure
Bin
Testar.bat
File to run Testar
Output
Sequences_ suspicioustitle
Sequences_ unexpectedclose
Graphs
Testar
Sequences that matched oracles
Sequences where the SUT closed unexpectedly
Contains raw graph files and a converter
See TA’s lab instructions for explanation.
SUTs
Note: This slide only shows the most important folders
Calculator
ATM
For task in lab
For homework task

13. Interpreting output (1)
User must determine if the reported failure is…
false positive – reported as failure, but is not really
true positive – an actual failure
That’s why the folders are called „unexpected…“ and „suspicious…“
User must look through all the sequences, even though many are duplicates, because there is no way to automatically analyse them

14. Interpreting output (2)
Testar generates files, which can be converted to visual graphs
„offline_graph_conversion.bat“ is automatically provided in each graph folder
All the graphs are converted automatically by the .bat file
We use the screenshotted graph to understand what happend in the sequence
Looking through the screenshotted graphs should be quite simple and easy to understand, however…
It can happen that there are some layout issues and not all the screenshots are connected

15. Like in this example, where you can see only one state before failure
Which was previous state?

16. To find the previous state, use the action number inside the brakets

17. Search for lower number to find previous state
Ctrl + f

18. It will show you the previous action
This way you can move through the graph if some parts are not connected

19. Hints (1)
Use a reasonable number of sequences (say, something between 20 and 100)
If the number is too small, you might not trigger any failure
If the number is too high, the execution might take too long and the (manual) analysis of the output files might take too long
Use a reasonable number of actions in each sequence
It will keep graphs smaller and therefore easier to read
About 20 should be fine, but it depends how you use the filters

20. Hints (2)
If you keep finding the same failure many times, you should consider filtering out the path leading to that failure
It will save you time while inspecting graphs
Also, make sure, that you don’t apply filter too early, otherwise you might miss some of the failures
Information messages are not to be considered failures if they match those listed in the appendixes A (in-lab task) and C (homework task)
Don’t write oracles matching these messages
Check out Testar manual for more information about the tool