1. A Methodology and a Framework for Test Case Generation
University of Geneva
L.Lúcio, L.Pedro, D.Buchs

2. Talk Overview (Our) Model-Based test case generation theory
based on the work of Bernot, Gaudel, Marre;
The user’s role in the test case generation framework;
Test execution;
The test generation machinery.
L.Lúcio, L.Pedro, D. Buchs

3. Specification-Based Test Generation
Program
(P) ╞
Exhaustive
Test Set (TSP) ╞O
Explain what the oracle is – procedure that decides about the success of a test
The two different languages – specification and test (traces)
╞ : program P satisfies (has the same semantics as) specification SP;
╞o: program P reacts according to test set TSP (as observed by an Oracle O).
L.Lúcio, L.Pedro, D. Buchs

4. Exhaustive test set - Definition
The exhaustive set of tests for a given specification can be formalized as:
TExhaustive= { formula,result |
formula  composition of input,output pairs
result = true if formula models a valid behavior
result = false if formula models an invalid behavior }
Mention that the test is successful if either result is true and the trace holds in the implementation, or result is false and the trace does not hold in the implementation (as observed by the oracle).
The exhaustive test set describes fully the expected semantics of the specification, including valid and invalid behaviors…
L.Lúcio, L.Pedro, D. Buchs

5. Pertinence and Practicability
According to the previous slide, the following formula holds:
(P╞ SP) <=> (P╞o TSP)
IF test set TSP is pertinent – valid and unbiased
valid – accepts only correct programs (P╞o TSP) => (P╞ SP)
unbiased – doesn’t reject correct programs (P╞ SP) => (P╞o TSP)
If a program is correct by the specification then it has to pass the tests (P╞ SP) => (P╞o TSP)
If a program passes the tests then it has to satisfy the specification (P╞o TSP) => (P╞ SP)
This is a theoretical justification to the correction of the framework
But, exhaustive test sets are not practicable in the real world (infinite testing time)…
L.Lúcio, L.Pedro, D. Buchs

6. Test Selection How do we do this?
We thus need a way of reducing the exhaustive (infinite) test
set to a test set that is practicable, while keeping pertinence…
How do we do this?
By stating hypotheses about the behavior of the program – the idea is to find good hypotheses that generalize correctly the behavior of the SUT!
L.Lúcio, L.Pedro, D. Buchs

7. Stating Hypotheses
Example: consider as SUT a (simplified) embedded controller for
a drink vending machine:
Drink
Vending
Machine
accept_coin
insert_coin
reject_coin
select_drink (X)
give_drink (X)
not_enough_money
eject_money
Drinks available: Coke (2 coins), Water (1 coin)
L.Lúcio, L.Pedro, D. Buchs

8. Stating Hypotheses (2)
Hypotheses 1: if the SUT works well for sequences of at least
3 operations, then the system works well (regularity);
AND
Hypotheses 2: if the system works well while choosing one
kind of drink, then it will work well for choosing all kinds (uniformity).
The engineer provides hypotheses about the functioning of the SUT
 <insert_coin, accept_coin>,
<select_drink(water), give_drink>,
<select_drink(water), not_enough_money>,
true 
Example test 1
 <select_drink(water), give_drink>,
<insert_coin, accept_coin>,
false 
Example test 2
L.Lúcio, L.Pedro, D. Buchs

9. Specification – Complementing human Hypotheses
Example: imagine that from the two hypotheses in the last but one slide the test selection mechanism has chosen Water. There are then 3 interesting valid behaviors (tests):
The buyer doesn’t insert enough coins, chooses Water and gets nothing;
The buyer inserts enough coins, chooses Water and gets it;
The buyer inserts too many coins, chooses Water, gets it and the change.
The richest the specification, the bigger the amount of automatically deduced hypotheses
The points of choice stated in the operation of the specification can be used to add further behavior classification that can be combined with the hypotheses stated by the test engineer. We call this sub-domain decomposition.
L.Lúcio, L.Pedro, D. Buchs

10. Test set Reduction Process
<H0, T0> Oracle hypothesis and exhaustive test set T0 H0
Reduce
Test
set
Increase
hypothesis T1 H1 … …
Tk-1
Hk-1 Tk Hk
<Hk, Tk> defined hypothesis and practicable test set
L.Lúcio, L.Pedro, D. Buchs

11. (Hypotheses H Application)
Full Picture
Does P satisfy SP? P SP
Test Selection
(Hypotheses H Application)
Test Set T
Test
Procedure
Execution of P using T
Correction of P
Oracle
P satisfies, or not, T! no
inconclusive
yes
P satisfies, or not, H no
yes
P does not satisfy SP
Undefined
P satisfies SP!
L.Lúcio, L.Pedro, D. Buchs

12. Full picture User’s Role in the Framework
Test validation +
Spec. hypotheses
Model
Tests
SUT
(implementation)
Selection
Criteria
Discussed later
in the presentation +
Hypotheses about
SUT behavior
Oracle
Test
Driver
Explain black box…
Test
Results
L.Lúcio, L.Pedro, D. Buchs

13. Behavior classification
Providing the Model…
Test Validation +
Behavior classification
Model
Tests
Selection
Criteria
Test
Driver
SUT
(implementation)
Oracle
Results +
Tests
SUT
(implementation)
Selection
Criteria +
Oracle
Test
Driver
Explain black box…
Test
Results
L.Lúcio, L.Pedro, D. Buchs

14. Providing the Model The Modelling Language
UML (Fondue specification)
Software development process for reactive systems
Purpose is to produce a description of:
The problem domain
The functional requirements of the system
What it provides
Concept Model: Static structure of the information in the system
Behavior Model: Input and Output communication of the system
Uses UML (Class / Collaboration / State) and OCL
Talk about the fact that we have tool suport for parsing Fondue…
Renaming of the diagrams to Concept, Environment, Protocol and Operation Models
L.Lúcio, L.Pedro, D. Buchs

15. Providing the Model Expressing Model Semantics
Describes operations on the system by operation schemas which specify operations by pre- and post conditions using OCL
System State
post-condition
OCL
Operation
Schema
For X
Operation X
System State
pre-condition
States of the system correspond to abstract representations
Output Event
State change implies changing objects, attributes and relations described by the class model …
L.Lúcio, L.Pedro, D. Buchs

16. Operation’s signatures
Model - SUT Mapping…
Operation’s signatures
SUT
Interface
Mapping
Model
Tests
Selection
Criteria
Oracle
Test
Results +
Tests
Selection
Criteria
Oracle
Test
Results +
SUT
(implementation)
Test
Driver
Explain black box…
L.Lúcio, L.Pedro, D. Buchs

17. Model – SUT Mapping The Test Driver
The Test Driver is the piece of software that allows executing (abstract) tests on the SUT;
Each model operation and operation output has to be connected with an SUT operation and output respectively;
Heterogeneous SUT interfaces:
Programmatic APIs
Web pages
Specialized graphical interfaces
Command line
L.Lúcio, L.Pedro, D. Buchs

18. Model – SUT Mapping Example
Model’s Operations
insert_coin
select_drink (X)
eject_money
Operations’ Output
accept_coin
reject_coin
give_drink (X)
not_enough_money
L.Lúcio, L.Pedro, D. Buchs

19. Providing the Oracle Hypotheses
Model
Tests
Selection
Criteria
SUT
(implementation)
Test
Results +
Model
Tests
Selection
Criteria
Test
Driver
SUT
(implementation)
Oracle
Results +
Explain black box…
Observability
hypotheses
Observes
L.Lúcio, L.Pedro, D. Buchs

20. Providing the Oracle Hypotheses Observing test results
The Oracle will observe the driver while it applies <input,output> pairs to the SUT;
If the output is not exactly what the expected one , the Oracle Hypotheses provide an error margin…
If the output goes outside the established hypotheses, the test result is undefined!
Example: An operation was performed on a web page
and after 40 seconds there is still no reply…
Example Oracle hypotheses: if in a test there is a delay
of more than 30 seconds then the test doesn’t pass.
L.Lúcio, L.Pedro, D. Buchs

21. Providing Hypotheses about the SUT
Model
Tests
Test
Driver
SUT
(implementation)
Oracle
Results +
Model
Tests
Test
Driver
SUT
(implementation)
Oracle
Results +
Model
Tests
SUT
(implementation)
Selection
Criteria +
Hypotheses about
SUT behavior
Oracle
Test
Driver
Explain black box…
Test
Results
L.Lúcio, L.Pedro, D. Buchs

22. Providing Hypotheses about the SUT The Hypotheses’ language
We are concentrating both the languages to:
Express hypotheses – or constraints on tests
Express tests in Prolog!...
Example: produce a set of tests where the user inserts coins and
selects a drink. The number of coins should be inferior to 5, testing
with one type of drink is enough and the behavior should be valid.
Mention that the hypotheses specification language includes and/or/not as well as * and +
Includes also uniform and exhaust
There is not a full connection with the theory (yet)…
tests(L) :- pattern(_,and(star(ev(insertMoney(1),_),N),ev(selectDrink(S),_)),L),
lowerT(nat,N,5),uniform(L),valid(L,true).
Result
with(selectDrink(Coke),[insufficientFunds])
with(insertMoney(1),[]), with(selectDrink(Coke),[insufficientFunds])
with(insertMoney(1),[]), with(insertMoney(1),[]), with(selectDrink(Coke), [giveDrink(Coke)])
L.Lúcio, L.Pedro, D. Buchs

23. Providing Hypotheses about the SUT The Hypotheses’ language (2)
Using Prolog we can now express hypotheses on:
action sequences:
Pattern of paths based on regular expressions:
Zero or more times the same operation(s): *
One or more times the same operation(s): +
Logical operators between operation sequences: AND,OR
variable value domains:
Explore all possible values of a given domain;
Regularity hypotheses:
Constrain the values of a domain.
Uniformity hypotheses:
Choose only one value belonging to a variable’s domain
List of path language not exhaustive
L.Lúcio, L.Pedro, D. Buchs

24. Test Translation - example
We have used a demo version of a test tool for
web pages – QuicktestTM
L.Lúcio, L.Pedro, D. Buchs

25. Test Execution
L.Lúcio, L.Pedro, D. Buchs

26. Test case generation Machinery
Tests
Selection
Criteria
Test
Driver
SUT
Oracle
Results +
Model
Tests
Selection
Criteria
Test
Driver
SUT
Oracle
Results +
Model
Syntactic test creation
Test validation
Subdomain decomposition
L.Lúcio, L.Pedro, D. Buchs

27. Operational specification model
Syntactically building tests while applying hypotheses isn’t a problem…
... but validating tests requires an operational specification model!
…also required to find sub-domains for operation’s parameters!
L.Lúcio, L.Pedro, D. Buchs

28. Operational specification model: UML→CO-OPN→Prolog
CO-OPN is a formal OO specification language based on:
ADTs : data types (no persistent state);
Petri Nets : behavior and concurrency handling.
Prolog allows constrained execution of the Model…
L.Lúcio, L.Pedro, D. Buchs

29. UML→CO-OPN→Prolog Translation Tools
We have already a tool that translates CO-OPN into an operational Prolog model!
UML->CO-OPN (translation from UML MOF to CO-OPN MOF)
Image
UML
CO-OPN
Prolog
Under
Construction
Exists
L.Lúcio, L.Pedro, D. Buchs

30. Subdomain decomposition with Prolog
We wish to find which are the possible behaviors of an operation;
Prolog’s backtracking mechanism is very useful for that task…
a: a1
PC: True
a: a1+1
(1)
PC: a1 > 9
PC: a1<=9
(2 -true)
(2 - false)
Int foo (int a) {
1 a++;
2 if (a>10) ;
4 else ;
6 return a }
backtracking
point
The constraints in PC (Path Condition) define a decomposition of
the domains of the operation’s entry parameters…
L.Lúcio, L.Pedro, D. Buchs

31. Conclusions The framework and the process seem to be in place;
We are working on the UML/COOPN translation;
Unified user interface for the framework needed;
Validation in industry needed;
Come up with domain specific hypotheses templates
L.Lúcio, L.Pedro, D. Buchs