A Methodology and a Framework for Test Case Generation University of Geneva L.Lúcio, L.Pedro, D.Buchs

Talk Overview (Our) Model-Based test case generation theory based on the work of Bernot, Gaudel, Marre; The user’s role in the test case generation framework; Test execution; The test generation machinery. L.Lúcio, L.Pedro, D. Buchs

Specification-Based Test Generation Program (P) ╞ Exhaustive Test Set (TSP) ╞O Explain what the oracle is – procedure that decides about the success of a test The two different languages – specification and test (traces) ╞ : program P satisfies (has the same semantics as) specification SP; ╞o: program P reacts according to test set TSP (as observed by an Oracle O). L.Lúcio, L.Pedro, D. Buchs

Exhaustive test set - Definition The exhaustive set of tests for a given specification can be formalized as: TExhaustive= { formula,result | formula  composition of input,output pairs result = true if formula models a valid behavior result = false if formula models an invalid behavior } Mention that the test is successful if either result is true and the trace holds in the implementation, or result is false and the trace does not hold in the implementation (as observed by the oracle). The exhaustive test set describes fully the expected semantics of the specification, including valid and invalid behaviors… L.Lúcio, L.Pedro, D. Buchs

Pertinence and Practicability According to the previous slide, the following formula holds: (P╞ SP) <=> (P╞o TSP) IF test set TSP is pertinent – valid and unbiased valid – accepts only correct programs (P╞o TSP) => (P╞ SP) unbiased – doesn’t reject correct programs (P╞ SP) => (P╞o TSP) If a program is correct by the specification then it has to pass the tests (P╞ SP) => (P╞o TSP) If a program passes the tests then it has to satisfy the specification (P╞o TSP) => (P╞ SP) This is a theoretical justification to the correction of the framework But, exhaustive test sets are not practicable in the real world (infinite testing time)… L.Lúcio, L.Pedro, D. Buchs

Test Selection How do we do this? We thus need a way of reducing the exhaustive (infinite) test set to a test set that is practicable, while keeping pertinence… How do we do this? By stating hypotheses about the behavior of the program – the idea is to find good hypotheses that generalize correctly the behavior of the SUT! L.Lúcio, L.Pedro, D. Buchs

Stating Hypotheses Example: consider as SUT a (simplified) embedded controller for a drink vending machine: Drink Vending Machine accept_coin insert_coin reject_coin select_drink (X) give_drink (X) not_enough_money eject_money Drinks available: Coke (2 coins), Water (1 coin) L.Lúcio, L.Pedro, D. Buchs

Stating Hypotheses (2) Hypotheses 1: if the SUT works well for sequences of at least 3 operations, then the system works well (regularity); AND Hypotheses 2: if the system works well while choosing one kind of drink, then it will work well for choosing all kinds (uniformity). The engineer provides hypotheses about the functioning of the SUT  <insert_coin, accept_coin>, <select_drink(water), give_drink>, <select_drink(water), not_enough_money>, true  Example test 1  <select_drink(water), give_drink>, <insert_coin, accept_coin>, false  Example test 2 L.Lúcio, L.Pedro, D. Buchs

Specification – Complementing human Hypotheses Example: imagine that from the two hypotheses in the last but one slide the test selection mechanism has chosen Water. There are then 3 interesting valid behaviors (tests): The buyer doesn’t insert enough coins, chooses Water and gets nothing; The buyer inserts enough coins, chooses Water and gets it; The buyer inserts too many coins, chooses Water, gets it and the change. The richest the specification, the bigger the amount of automatically deduced hypotheses The points of choice stated in the operation of the specification can be used to add further behavior classification that can be combined with the hypotheses stated by the test engineer. We call this sub-domain decomposition. L.Lúcio, L.Pedro, D. Buchs

Test set Reduction Process <H0, T0> Oracle hypothesis and exhaustive test set T0 H0 Reduce Test set Increase hypothesis T1 H1 … … Tk-1 Hk-1 Tk Hk <Hk, Tk> defined hypothesis and practicable test set L.Lúcio, L.Pedro, D. Buchs

(Hypotheses H Application) Full Picture Does P satisfy SP? P SP Test Selection (Hypotheses H Application) Test Set T Test Procedure Execution of P using T Correction of P Oracle P satisfies, or not, T! no inconclusive yes P satisfies, or not, H no yes P does not satisfy SP Undefined P satisfies SP! L.Lúcio, L.Pedro, D. Buchs

Full picture User’s Role in the Framework Test validation + Spec. hypotheses Model Tests SUT (implementation) Selection Criteria Discussed later in the presentation + Hypotheses about SUT behavior Oracle Test Driver Explain black box… Test Results L.Lúcio, L.Pedro, D. Buchs

Behavior classification Providing the Model… Test Validation + Behavior classification Model Tests Selection Criteria Test Driver SUT (implementation) Oracle Results + Tests SUT (implementation) Selection Criteria + Oracle Test Driver Explain black box… Test Results L.Lúcio, L.Pedro, D. Buchs

Providing the Model The Modelling Language UML (Fondue specification) Software development process for reactive systems Purpose is to produce a description of: The problem domain The functional requirements of the system What it provides Concept Model: Static structure of the information in the system Behavior Model: Input and Output communication of the system Uses UML (Class / Collaboration / State) and OCL Talk about the fact that we have tool suport for parsing Fondue… Renaming of the diagrams to Concept, Environment, Protocol and Operation Models L.Lúcio, L.Pedro, D. Buchs

Providing the Model Expressing Model Semantics Describes operations on the system by operation schemas which specify operations by pre- and post conditions using OCL System State post-condition OCL Operation Schema For X Operation X System State pre-condition States of the system correspond to abstract representations Output Event State change implies changing objects, attributes and relations described by the class model … L.Lúcio, L.Pedro, D. Buchs

Operation’s signatures Model - SUT Mapping… Operation’s signatures SUT Interface Mapping Model Tests Selection Criteria Oracle Test Results + Tests Selection Criteria Oracle Test Results + SUT (implementation) Test Driver Explain black box… L.Lúcio, L.Pedro, D. Buchs

Model – SUT Mapping The Test Driver The Test Driver is the piece of software that allows executing (abstract) tests on the SUT; Each model operation and operation output has to be connected with an SUT operation and output respectively; Heterogeneous SUT interfaces: Programmatic APIs Web pages Specialized graphical interfaces Command line L.Lúcio, L.Pedro, D. Buchs

Model – SUT Mapping Example Model’s Operations insert_coin select_drink (X) eject_money Operations’ Output accept_coin reject_coin give_drink (X) not_enough_money L.Lúcio, L.Pedro, D. Buchs

Providing the Oracle Hypotheses Model Tests Selection Criteria SUT (implementation) Test Results + Model Tests Selection Criteria Test Driver SUT (implementation) Oracle Results + Explain black box… Observability hypotheses Observes L.Lúcio, L.Pedro, D. Buchs

Providing the Oracle Hypotheses Observing test results The Oracle will observe the driver while it applies <input,output> pairs to the SUT; If the output is not exactly what the expected one , the Oracle Hypotheses provide an error margin… If the output goes outside the established hypotheses, the test result is undefined! Example: An operation was performed on a web page and after 40 seconds there is still no reply… Example Oracle hypotheses: if in a test there is a delay of more than 30 seconds then the test doesn’t pass. L.Lúcio, L.Pedro, D. Buchs

Providing Hypotheses about the SUT Model Tests Test Driver SUT (implementation) Oracle Results + Model Tests Test Driver SUT (implementation) Oracle Results + Model Tests SUT (implementation) Selection Criteria + Hypotheses about SUT behavior Oracle Test Driver Explain black box… Test Results L.Lúcio, L.Pedro, D. Buchs

Providing Hypotheses about the SUT The Hypotheses’ language We are concentrating both the languages to: Express hypotheses – or constraints on tests Express tests in Prolog!... Example: produce a set of tests where the user inserts coins and selects a drink. The number of coins should be inferior to 5, testing with one type of drink is enough and the behavior should be valid. Mention that the hypotheses specification language includes and/or/not as well as * and + Includes also uniform and exhaust There is not a full connection with the theory (yet)… tests(L) :- pattern(_,and(star(ev(insertMoney(1),_),N),ev(selectDrink(S),_)),L), lowerT(nat,N,5),uniform(L),valid(L,true). Result with(selectDrink(Coke),[insufficientFunds]) with(insertMoney(1),[]), with(selectDrink(Coke),[insufficientFunds]) with(insertMoney(1),[]), with(insertMoney(1),[]), with(selectDrink(Coke), [giveDrink(Coke)]) L.Lúcio, L.Pedro, D. Buchs

Providing Hypotheses about the SUT The Hypotheses’ language (2) Using Prolog we can now express hypotheses on: action sequences: Pattern of paths based on regular expressions: Zero or more times the same operation(s): * One or more times the same operation(s): + Logical operators between operation sequences: AND,OR variable value domains: Explore all possible values of a given domain; Regularity hypotheses: Constrain the values of a domain. Uniformity hypotheses: Choose only one value belonging to a variable’s domain List of path language not exhaustive L.Lúcio, L.Pedro, D. Buchs

Test Translation - example We have used a demo version of a test tool for web pages – QuicktestTM L.Lúcio, L.Pedro, D. Buchs

Test Execution L.Lúcio, L.Pedro, D. Buchs

Test case generation Machinery Tests Selection Criteria Test Driver SUT Oracle Results + Model Tests Selection Criteria Test Driver SUT Oracle Results + Model Syntactic test creation Test validation Subdomain decomposition L.Lúcio, L.Pedro, D. Buchs

Operational specification model Syntactically building tests while applying hypotheses isn’t a problem… ... but validating tests requires an operational specification model! …also required to find sub-domains for operation’s parameters! L.Lúcio, L.Pedro, D. Buchs

Operational specification model: UML→CO-OPN→Prolog CO-OPN is a formal OO specification language based on: ADTs : data types (no persistent state); Petri Nets : behavior and concurrency handling. Prolog allows constrained execution of the Model… L.Lúcio, L.Pedro, D. Buchs

UML→CO-OPN→Prolog Translation Tools We have already a tool that translates CO-OPN into an operational Prolog model! UML->CO-OPN (translation from UML MOF to CO-OPN MOF) Image UML CO-OPN Prolog Under Construction Exists L.Lúcio, L.Pedro, D. Buchs

Subdomain decomposition with Prolog We wish to find which are the possible behaviors of an operation; Prolog’s backtracking mechanism is very useful for that task… a: a1 PC: True a: a1+1 (1) PC: a1 > 9 PC: a1<=9 (2 -true) (2 - false) Int foo (int a) { 1 a++; 2 if (a>10) 3 ...; 4 else 5 ...; 6 return a } backtracking point The constraints in PC (Path Condition) define a decomposition of the domains of the operation’s entry parameters… L.Lúcio, L.Pedro, D. Buchs

Conclusions The framework and the process seem to be in place; We are working on the UML/COOPN translation; Unified user interface for the framework needed; Validation in industry needed; Come up with domain specific hypotheses templates L.Lúcio, L.Pedro, D. Buchs