1. 19.09.2002 Intel Academic Forum. Budapest, September, 2002 ISPRAS Experience in Model Based Testing Alexander K. Petrenko, Institute for System Programming of Russian Academy of Sciences (ISPRAS), http://www.ispras.ru

2. 19.09.2002 Intel Academic Forum. Budapest, September, 2002 ISPRAS Experience in Industrial Model Based Testing

3. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Why Model Based Testing? Exhaustive testing that covers all implementation paths is impossible. Exhaustive implementation based (“white box”) testing does not guaranty correct functionality. White box testing leads to increasing duration of development because test development can be launched only when the implementation is completed. Nevertheless, we like to conduct systematic testing. Formal models propose basis for systematic testing, we derive from the models  test coverage metrics,  input stimulus,  results correctness criteria. test development ahead of implementation schedule.

4. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Model Checking vs. Model Based Testing Model CheckingModel Based Testing Answers the question Is the model correct?Does the implementation behavior conform to the model behavior? Expected resultCorrect modelTest suite for the implementation testing and proper implementation Complexity of the models More simple in comparison with implementation because restriction of analytic analysis Close to complexity of implementation under test Relation between model and implementation Very complicatedSimple

5. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Synonyms Models (Formal) Specification We consider behavior/functional models. The models provide simplified, abstract view on the target software/hardware. Processing of the models needs their formal description/specification.

6. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Model Based Testing Approach Generate exhaustive test suites for model of implementation Translate the test suites to implementation level Apply the tests to implementation under test (Optionally) Interpret the testing results in terms of the model

7. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Related Works IBM Research Laboratory (Haifa, Israel) Microsoft Research (Redmond, US)

8. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Examples of Model Based Testing Applications IBM Research Laboratory (Haifa, Israel)  Store Date Unit – digital signal processor  API of file system, telephony and Internet protocols etc. Microsoft Research (Redmond, US)  Universal PnP interface ISPRAS (Moscow, Russia)  Kernel of operating system (Nortel Networks)  IPv6 protocol (Microsoft)  Compiler optimization units (Intel)  Massive parallel compiler testing (RFBR, Russia)

9. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Origin of ISPRAS Methods 1987-1994 Test suite for compiler of real-time programming language for “Buran” space shuttle 1994 – 1996 ISP RAS – Nortel Networks contract on functional test suite development for Switch Operating System kernel  Few hundreds of bugs found in the OS kernel, which had been 10 years in use KVEST technology About 600K lines of Nortel code tested by 2000

10. Intel Academic Forum. Budapest, September, 2002 19.09.2002 ISPRAS Model Based Testing: Two Approaches UniTesK Testing of Application Program Interfaces (API) based on Software Contract Lama Compiler testing based on LAnguage Model Application (Lama)

11. 19.09.2002 Intel Academic Forum. Budapest, September, 2002 UniTesK Testing of Application Program Interfaces (API)

12. Intel Academic Forum. Budapest, September, 2002 19.09.2002 What is API? User Interface Application Program Interface (API)

13. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Functional Testing UniTesK method deals with functional testing Requirements Formal Specifications Tests To automate testing we provide a formal representation of requirements

14. Intel Academic Forum. Budapest, September, 2002 19.09.2002 UniTesK Process Phases Techniques Pre- and post-conditions, invariants Implicit Finite State Machines (FSM), data iterators Test coverage metrics based on specification structure Interface specification Test scenario description Test execution Test result analysis

15. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Decomposition of Testing Tasks The entire test is a test sequence intended to achieve specified coverage From specification we can generate oracles and define test coverage metrics Test sequence construction Test oracles System under test

16. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Test Suite Architecture Legend: Automatic derivation Pre-builtManualGenerated SpecificationTest coverage tracker Test oracle Data model System under test Mediator Java/C/C++/C# mediator Test scenarioScenario driverTest engine

17. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Test Oracle Specification of method f integer f (a : float) post { post_f (a, f_result) } Test Oracle for the method f f_result = f(x) post_f(x,f_result) verdict = true verdict = false

18. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Test Coverage Metrics Based on Specification Structure Specification post { if (a || b || c || d && e ) { branch “OK“;..... } else { branch “Bad parameters" ;..... } } Partition (Derivation of branches and logical terms) BRANCH “OK”  a -- op1  !a && b -- op2  !a && !b && c -- op3 ... BRANCH “Bad parameters"  !a && !b && !c && !d  !a && !b && !c && d && !e

19. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Test Sequence Generation We use FSM to generate test sequences which traverse all equivalence classes defined by partition analysis. S1 S4 S2 S3 op2 op3 op2 op1 op3 But full FSM description is a labor consuming and tedious work.

20. Intel Academic Forum. Budapest, September, 2002 19.09.2002 SC1 SC4 SC2 SC3 Equivalence classes of states FSM Construction. Statics SC1 SC4 SC2 SC3 op2 op3 op2 op1 op3 Partition (Branches and logical terms) BRANCH “OK”  a -- op1  !a && b -- op2  !a && !b && c-- op3 ... BRANCH "Bad parameters"  !a && !b && !c && !d -– op i  !a && !b && !c && d && !e -– op i+1 Partition (Branches and logical terms) BRANCH “OK”  a -- op1  !a && b -- op2  !a && !b && c-- op3 ... BRANCH "Bad parameters"  !a && !b && !c && !d -– op i  !a && !b && !c && d && !e -– op i+1 First step of FSM construction: -state and transition partition based on pre- and post-condition structure (FSM factorization) -test input iterators

21. Intel Academic Forum. Budapest, September, 2002 19.09.2002 FSM Construction. Dynamics Second step of FSM construction SC1 SC4 SC2 SC3 op2 1 op3 op2 op1 op3 SC1 SC4 SC2 SC3 op2 op3 op2 op1 op3 Result of test execution

22. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Model Based Testing: problems of deployment 1994 – 1996 ISP RAS – Nortel Networks contract on functional test suite development for Switch Operating System kernel  Few hundreds of bugs found in the OS kernel, which had been 10 years in use KVEST technology About 600K lines of Nortel code tested by 2000 KVEST had been deployed only in Nortel’s regression testing process. Why? Only few formal techniques used in real life practice. Why?

23. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Problems of Model Based Testing Deployment ProblemUniTesK solution Formal models for analytical verification are too simple for test generation. Ratio of models (specifications) and implementation size is about 1:5-10. Mediators provide a bridge between abstract models and implementation. Executable models cannot provide test oracles in common case because dependence on implementation and indeterminism. Implicit specifications (pre- and post-conditions) provide the test oracles. Test sequence generation needs very huge models (for example, like FSM). Implicit FSM. Usual number of states is about 5- 20. How to estimate test quality without implementation test coverage? Structure of pre- and post-condition is very informative and quite simple for the test coverage metrics. Gap between formal techniques and software/hardware development practice. Usual programming languages are extended for specification purpose.

24. Intel Academic Forum. Budapest, September, 2002 19.09.2002 UniTesK Tools and Applications CTesK – C testing tool -- alpha version  Microsoft IPv6 implementation J@T – Java testing tool -- beta version  Partially tested by itself  API of parallel debugger of mpC IDE (mpC is a parallel extension of C)  Posix/Win32 File I/O subsystem VDM++TesK -- free Further steps: C#TesK and C++TesK, (conceivably) VHDLTesK

25. 19.09.2002 Intel Academic Forum. Budapest, September, 2002 Lama Compiler testing based on Language Models Application

26. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Pilot project under contract with Intel Model Based Testing of Compiler Optimization Units Automate optimization unit test generation: Improve test coverage of the units Automate test oracle problem

27. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Lama approach Lama stands for Compiler testing based on LAnguage Models Application. Lama process steps: Given: a programming language (PL) Invent a model (simplified) language (ML) of PL Generate a set of test “programs” in the ML Map ML test “programs” into PL test programs Run compiler (or a compiler unit) to process test program in PL and analyze correctness of the compiler results

28. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Process of optimization unit testing Model language building blocks Faults & test coverage reports Optimization background Program Language (PL) Specification Model language design Step 1 Iterator development Step 2 Mapper development Step 3 Step 4 Test “programs” in ML Test programs in PL Test Execution & Test result analysis

29. Intel Academic Forum. Budapest, September, 2002 19.09.2002 An Example: Common Subexpression Elimination Optimization Label... Instruction Transition to label Basic block IF instruction if conditionthen blockelse block Common subexpression Step 1

30. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Result of translation into C Step 3 if ( (('c' - 'a') + (('c' - 'a') > ('c' - 'a'))) ) { (('c' - 'a') + (('c' - 'a') < ('c' - 'a'))); } else { (('c' - 'a') + (('c' - 'a') >= ('c' - 'a'))); }

31. 19.09.2002 Intel Academic Forum. Budapest, September, 2002 Conclusion

32. Intel Academic Forum. Budapest, September, 2002 19.09.2002 Conclusion on UniTesK&Lama Both UniTesK and Lama follow model based testing approach Base idea: Testing complex software by means of exhaustive coverage of relatively simple models Area of applicability: Any software and hardware components with well-defined interfaces or functional properties

33. Intel Academic Forum. Budapest, September, 2002 19.09.2002 References 1. A. K. Petrenko, I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. UniTesK Test Suite Architecture// Proceedings of FME’2002 conference, Copenhagen, Denmark, LNCS, No. 2391, 2002, pp. 77-88. 2. A.Petrenko. Specification Based Testing: Towards Practice// VI Ershov conference proc., LNCS 2244, 2001. 3. A. K. Petrenko, Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Experiences in using testing tools and technology in real-life applications. Proceedings of SETT’01, Pune, India, 2001. 4. I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin. Using Finite State Machines in Program Testing// Programming and Computer Software, Vol. 26, No. 2, 2000, pp. 61-73 (English version). 5. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications// Proceedings of World Congress of Formal Methods, Toulouse, France, LNCS, No. 1708, 1999, pp. 608-621. 6. I. B. Bourdonov, A. S. Kossatchev, V. V. Kuliamin, A. V. Maximov. Testing Programs Modeled by Nondeterministic Finite State Machine. (www.ispras.ru/~RedVerst/, white papers).www.ispras.ru/~RedVerst/