1. Test Oracle Generation Procedures and Executable Specifications 4 th Annual International Software Testing Conference 2004 Authors: Rupak Das and Sita Ramakrishnan Presented by: Dr Sita Ramakrishnan

2. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 2 Overview Oracle – an example A Generic Test Environment Test Oracles and their Importance Problems Assertion-based Implementation Oracles Specification Oracles Real Specification Languages Testability and Diagnosability with industry standards AsmL Oracles and the Future Conclusions

3. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 3 Oracle – an example The Oracle in Matrix Is a program Predicts choices and outcomes based on input from the Zion mainframe Test Oracles … The Oracle offers Candy to Neo, the following dialogue ensues … Neo: D'you already know if I'm going to take it? The Oracle: Wouldn't be much of an Oracle if I didn't.

4. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 4 A Generic Test Environment

5. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 5 A Generic Test Environment 3 main aspects to the test environment: 1.The test model, which is the application model is augmented to include testing requirements - By adding application specific constraints to the component interface protocol 2.The test model (state model) is fed to the test design software

6. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 6 A Generic Test Environment This software tool uses the critical properties of testing requirements built in the test model to generate automated test sequences 3. The test execution phase is the diagnosis phase. Accepts test inputs from SUT, and test is evaluated as pass or fail.

7. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 7 Testing Environment in VISWAS oOur Testing Environment in VISWAS supports such an automated process oIncludes the Test Sequence Generation phase and the System under Test

8. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 8 Testing Environment in VISWAS oDiagnosability was not the focus of concern in VISWAS, and test execution phase of test environment was not automated in VISWAS oNext we look at Test Execution phase and Test Oracle Generation Procedures

9. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 9 Test Oracles – What are they? A method to establish that the system under test has produced expected outputs corresponding to given test inputs. All testing techniques assume the existence of an oracle, be it a human `eyeball oracle, or a software oracle with valid input/output pairs. Valid input/output pairs (automatically) generated from specifications helps to verify `intended behaviour of the system.

10. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 10 Active and Passive Oracles Active Oracles mimic the behaviour of the software under test mainly built for accuracy rather than efficiency Passive Oracles Verifies software behaviour without replicating it

11. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 11 The Problems Reliable test set problem Whole input domain needs to be taken into account Otherwise it becomes impossible to create a reliable exhaustive test set Oracle problem Grey areas between intended behaviour, expected behaviour and actual behaviour Some opine that `truly general oracles may not be achieved

12. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 12 Assertion-based Implementation Oracles Assertions are ways to specify what a system is supposed to do, and not how it is to do it Embedded assertions in source code can be used to create implementation oracles Not a dedicated specification mechanism Pre-processing of code is often required

13. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 13 Assertion-based Implementation Oracles … Assertions AsmL specification of a stack interface

14. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 14 Assertion-based Implementation Oracles … Anna (ANNotated Ada) APP (Annotation Pre-Processor) for C Nana (Assertion library for C/C++) Eiffel Built in assertion support Integrated with exception-handling Java Multitude of assertion systems iContract, JaWa, Handshake, JMSAssert, jContractor and JUnit

15. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 15 Assertion-based Implementation Oracles Dependency on control points Assertions are treated as program statements with pre-defined execution points Caching of values Pre- and Post-Condition pairs relate the program states before and after method execution Expensive for complex/large data structures Existential and Universal Quantification Translating input to reflect quantification in terms of `loops of execution can be cumbersome

16. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 16 Specification Oracles Translating a specification into an oracle irrespective of implementation Oracles can be generated automatically from specifications or documentation Can extract relevant input/output pairs of test data from specification to check intended behaviour

17. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 17 Specification Oracles … Program Testing Assistant Defines test cases interactively Uses layer correspondences to generate test cases automatically from a library of templates Test cases execute when appropriate Success criteria defined by specifier Requires at least one successful execution to generate valid test data sets Chapman 1982, A Programmers Testing Assistant

18. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 18 Specification Oracles … Test Oracle Generator (TOG) Automatically generates test oracles from a relational specification that uses tabular expressions Test harness (oracle) takes input/output pair from Program Under Test (PUT) Relation is evaluated Returns true/false Peters and Parnas 1994, Test Oracle Generator

19. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 19 Specification Oracles … Documentation components required by TOG Program Specification Auxiliary Predicates and Functions User Definitions A relational specification is created If result of TOG execution is present in the relational specification table, oracle procedure execution is successful

20. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 20 Specification Oracles TOG oracles are only as good as the specification Access to data structures required to evaluate expressions and predicates Complexity of test harness reduces readability of the oracle Developed further by Software Engineering Research Group at McMaster University, Ontario

21. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 21 Real Specification Languages Actual specifications are translated to oracles for validation and verification Generate test oracles from specifications that are executable in their own right B, Z, AsmL (Abstract State Machine Language) Deriving test oracles require an ``executable subset of the original specification

22. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 22 Real Specification Languages Automatic Test Result Evaluation approach – translates Z schema to an evaluation function Test Template Framework derives ``oracle templates from model based specification written in Z Multiple oracles from multi-language specifications composed as a unit

23. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 23 Testability and Diagnosability with industry standards oDiagnostic components constructed according to standards facilitate competition in the market place in terms of risks, cost and quality oStandards imply a maturity in the underlying technology, thus adding to the level of confidence. TTCN-3 is the only standardised language for the specification & implementation of test cases.

24. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 24 Testability and Diagnosability with industry standards oUML is a widely used industry standard for software component modelling. oGraphical format for TTCN-3 (GFT) is based on Message Sequence Charts (MSCs) and UML. oGFT extends TTCN-3 with test specific concepts such as verdicts & defaults. oGFT is the basis for the definition of UML 2.0 test profile.

25. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 25 Testability and Diagnosability with industry standards oUML 2.0 Test Profile (UTP) has the notion of an arbiter, which is a test component aimed at separating test behaviour from test evaluation oEvaluates test results and assigns verdicts of a test case o A verdict is the outcome of a test case being pass, fail, inconc or error as defined in TTCN-3

26. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 26 Testability and Diagnosability with industry standards oA Test case may have 4 outcomes: oPass, Fail, inconclusive or abort oDuring the execution of a test case, a test trace is generated and stored in a test log oUML 2.0 Test Profile, GFT, !TeLa! have similar diagnostic features

27. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 27 AsmL Oracles Abstract State Machine approach has been proven suitable for large-scale executable specifications AsmL is a high-level language that implements the ASM paradigm AsmL specifications can be simulated Microsoft Research - Foundations of Software Engineering Group

28. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 28 AsmL Oracles … Specification is a non-deterministic ASM program written in AsmL that has a fixed initial state Individual finite states are grouped together to form `hyperstates Results in a finite automaton – used to generate test suite Extraction of an FSM from an ASM specification by extracting algorithm

29. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 29 … and the future Non-deterministic finite state machines cannot be generated from ASMs by using the extraction algorithm Combination of the Test Template Framework and Automatic Test Result Evaluation approach for Z specifications could be used AsmL as a specification and implementation language can be used to generate exhaustive executable oracles AsmL oracles can be translated to oracles for other.NET aware languages

30. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 30 Conclusions Producing a comprehensive, precise and understandable specification is difficult enough Adding computability to specifications to generate oracles makes it even more harder Intrinsic dependence of testing on oracles make them essential Pros and Cons exist for all approaches Any of the existing approaches is better and less expensive compared to a human oracle More research and development of automated oracle generation tools will make the V & V process much easier and productive

31. 4th International Software Testing Conference Pune, India 20-21 Feb 2004 31 Contact Details Authors: Rupak Das and Sita Ramakrishnan Presenter: Dr Sita Ramakrishnan School Computer Science & Software Engineering Monash University Australia sitar@csse.monash.edu.au ph: 61-3-99052485 fax: 61-3-99031077