1. Towards Automatic Generation of Parameterized Test Cases from Abstractions Jens R. Calamé Natalia Ioustinova Jaco van de Pol Centrum voor Wiskunde en Informatica, Amsterdam

2. 20-04-2006IPA Lentedagen 20062 Agenda 1.Testing Theory 2.Data Abstraction 3.Test Case Parameterization 4.Some Lemmas 5.Conclusion

3. 20-04-2006IPA Lentedagen 20063 Testing Theory Conformance Testing: –Implementation Imp conforms specification Spec, iff for all traces t in Spec holds: all input should be accepted after t, but at most the specified output should be generated by Imp. –Based on Tretmans‘ ioco theory Test Generation by the tool TGV (Test Generation with Verification Techniques) –Based on state enumeration –Limitation: Data often leads to state-space explosion –Solution: Data abstraction mitigates this limitation

4. 20-04-2006IPA Lentedagen 20064 Our Test Generation Process Abstract System Specification System Specification (formal) Test Purpose TGV System Specification (e.g. UML) Data Abstraction Abstract Test Case Test Case Generation Parameterizable Test Case (TTCN-3) Rule System for Data Selection TTCN-3 Generation Constraint Generation

5. 20-04-2006IPA Lentedagen 20065 Data Abstraction Motivation: Input and output data from large (infinite) domains leads to state explosion  Application of state-based test generation tools impossible Solution (from model-checking): data abstraction –Introduction of a chaotic value ╥ for each datatype D –Lifting functions for original values from D –Replacement of input variables by ╥ and propagation through the system –Introduction of may functions for guards over three- value-logic (semantics: may( ╥ ) = true) –Consequence: Introduction of extra traces by non-determinism

6. 20-04-2006IPA Lentedagen 20066 Data Abstraction – Example ?getPin(x) ?getBalance 2 3 4 5 !Balance(b) 6 7 ?getAmount(y) ? (y) 8 01 ?initPin(p)?initBalance(b) !pinIncorrect  )(px  !pinCorrect  )(px  !Money(y); b:=b-y  )(by  !LowBalance(b)  )(by 

7. 20-04-2006IPA Lentedagen 20067 Data Abstraction – Test Case by TGV

8. 20-04-2006IPA Lentedagen 20068 Test Case Parameterization To be solved after data abstraction: –Pruning of traces introduced by overapproximation –Finding possible value ranges for test data Rule System (Prolog) –Represents SUT specification –Defines rules on data (addition, substraction; and, or etc.) –Defines rules for process behavior of Spec Query (Prolog) –Represents the abstract test case (one rule per trace) –All transitions in trace taken as query body –Alternative: one query for whole CTG

9. 20-04-2006IPA Lentedagen 20069 Testing with Abstraction – Rule System … getPin(state(2,P,B,X,Y), state(3,P,B,X1,Y), param(X1)). pinIncorrect(state(3,P,B,X,Y), state(8,P,B,X,Y),param(_)) :- X ≠ P. pinCorrect(state(3,P,B,X,Y), state(4,P,B,X,Y), param(_)) :- X = P. … ?getPin(x) ?getBalance 2 3 4 5 !Balance(b) 6 7 ?getAmount(y) ? (y) 8 01 ?initPin(p)?initBalance(b) !pinIncorrect  )(px  !pinCorrect  )(px  !Money(y); b:=b-y  )(by  !LowBalance(b)  )(by 

10. 20-04-2006IPA Lentedagen 200610 Testing with Abstraction – Query oracle(P,B,X,Yin,Yout) :- initPin(state(0,0,0,0,0), G1, param(P)), initBalance(G1, G2, param(B)), getPin(G2, G3, param(X)), pinCorrect(G3, G4, _), getAmount(G4, G5, param(Yin)), money(G5, _, param(Yout)). P=P{-1.0inf..1.0inf} B=B{-1.0inf..1.0inf} X=P{-1.0inf..1.0inf} Yin=Yout{-1.0inf..1.0inf} Yout=Yout{-1.0inf..1.0inf} Yout{-1.0inf..1.0inf}-B{-1.0inf..1.0inf}=<0

11. 20-04-2006IPA Lentedagen 200611 Test Execution 1.Pre-solve one trace to pass statically (skip internal steps) 2.Execute this trace until the SUT leaves it 3.Try to find a trace to pass  solve and execute it 4.If no trace to pass: Try to find a trace to inconc  solve it 5.If no trace to inconc: set test verdict fail 2 3 4 5 6 7PASS 01 INCONC !initPin(P)!initBalance(B) !getPin(X) ?pinCorrect ?pinInCorrect ?lowBalance !getAmount(Yin)!getBalance ?Balance(x50) !getAmount(x60) ?Money(Yout) ?eatCard

12. 20-04-2006IPA Lentedagen 200612 Some Lemmas The original system Spec is simulated by Spec ╥ in all details (and some more). The control flow of the synchronous product Spec x TP is simulated by Spec ╥ x TP. The set of accepting/refusing traces of Spec x TP is a subset of those of Spec ╥ x TP. If the test oracle holds for a trace in the CTG under a certain valuation, this trace is valid in the original system under the given valuation. The test algorithm terminates with a sound verdict. For further details and proofs see the technical report.

13. 20-04-2006IPA Lentedagen 200613 Case Study CEPS Common Electronic Purse Specifications: protocol for electronic payment using a multi-currency smart-card Input and output parameters of card actions mainly natural numbers  (in)finite domain Variables partially arrays (up to 16 elements in simplified µCRL realization) As µCRL specification: –54 summands –44 process variables (netto) –207process variables (brutto), due to arrays of structures

14. 20-04-2006IPA Lentedagen 200614 Case Study CEPS (cont'd) Instantiation and reduction (abstracted specification): ca. 16 min. on five 2.2GHz Athlon 64bit single CPU computers (1 GB RAM each) Generation of two test cases: 594 and 109 states, resp. in less than one second (one 2.2GHz AMD Athlon XP 32 bit CPU and 1 GB RAM) Constraint solving produces results in negligable time

15. 20-04-2006IPA Lentedagen 200615 Conclusion Data abstraction makes state-based test generation applicable to systems with large data domains Approach successfully evaluated on CEPS case study Ongoing and future work: –Redesign of tools (add "real" constraint-solving) –On-the-fly constraint-solving –Treatment of  -steps –(Generation and) execution of TTCN-3 test cases –Integration of UML as specification language

16. 20-04-2006IPA Lentedagen 200616 Related Links TT-Medal Project: www.tt-medal.orgwww.tt-medal.org Test generator TGV: –www-verimag.imag.fr/~async/TGVwww-verimag.imag.fr/~async/TGV –www.inrialpes.fr/vasy/cadpwww.inrialpes.fr/vasy/cadp Data abstraction tools: www.cwi.nl/~calame/dataabstr.html www.cwi.nl/~calame/dataabstr.html Technical report and papers: www.cwi.nl/~calame/works.html www.cwi.nl/~calame/works.html Jens R. Calamé jens.calame@cwi.nl Natalia Ioustinova ustin@cwi.nl Jaco van de Pol vdpol@cwi.nl