1. Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles Christian Murphy, Kuang Shen, Gail Kaiser Columbia University

2. Chris Murphy – Columbia University2 Problem Statement Some applications (e.g. machine learning, simulation) do not have test oracles that indicate whether the output is correct for arbitrary input Oracles may exist for a limited subset of the input domain, and gross errors (e.g. crashes) can be detected with certain inputs or other techniques However, it is difficult to detect subtle (computational) errors for arbitrary inputs

3. Chris Murphy – Columbia University3 Observation Even when there is no oracle in the general case (i.e. knowing the relationship between a particular input and its output), it may be possible to know relationships between sets of inputs and their corresponding outputs One such approach that takes advantage of this is “Metamorphic Testing” [Chen ’98]

4. Chris Murphy – Columbia University4 Metamorphic Testing An approach for creating follow-up test cases based on those that have not revealed any defects If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce a new test case, t(x) We can then predict the expected value of f(t(x)) based on the value of f(x) obtained from the actual execution

5. Chris Murphy – Columbia University5 Metamorphic Testing Example Consider a function std_dev(A[ ]) If we permute the elements in A, the result should stay the same If we multiply each element in A by 2, the result should be doubled These properties can be used to create a “pseudo-oracle” for the function

6. Chris Murphy – Columbia University6 Metamorphic Testing without an Oracle When a test oracle exists, we can know whether f(t(x)) is correct  Because we have an oracle for f(x)  So if f(t(x)) is as expected, then it is correct When there is no test oracle, f(x) acts as a “pseudo-oracle” for f(t(x))  If f(t(x)) is as expected, it is not necessarily correct  However, if f(t(x)) is not as expected, either f(x) or f(t(x)) (or both) is wrong

7. Chris Murphy – Columbia University7 Our Solution To address the problem of applications that have no test oracle, we use Metamorphic Testing at the function level To specify the metamorphic properties, we use extensions to the Java Modeling Language (JML) and a pre-processor called Corduroy To automate the process and address the need for initial input values, we check the properties at runtime

8. Chris Murphy – Columbia University8 Related Work Applying Metamorphic Testing to “non-testable programs” (applications without test oracles)  Chen IST’02, Chen ISSTA’02, Chan JWSR’07 Checking algebraic specifications at runtime  Sankar ISSTA’91, Nunes ICFEM’06 Checking program invariants at runtime  Flanagan FME’01, Hangal ICSE’02

9. Chris Murphy – Columbia University9 JML Basics Behavioral interface specification language that embraces the Design by Contract approach Developers can specify:  preconditions (“requires”)  postconditions (“ensures”)  arbitrary assertions Specifications can be checked dynamically using JML Runtime Assertion Checking tools

10. Chris Murphy – Columbia University10 Process Specify functions’ metamorphic properties using extension to JML Specify functions’ metamorphic properties using extension to JML Methods that check metamorphic properties Methods that check metamorphic properties Pure JML specifications that invoke test methods Pure JML specifications that invoke test methods Corduroy

11. Chris Murphy – Columbia University11 Metamorphic Properties in JML /*@ @ensures \result <= 1 && \result >= -1; @meta sine(x + 2 * PI) == \result @meta -1 * sine(-x) == \result */ public double sine (double x) {... }

12. Chris Murphy – Columbia University12 Extension to JML: Array Functions /*@ @ensures \result >= 0; @meta std_dev(\shuffle(A)) == \result @meta std_dev(\multiply(A, 2)) == \result * 2 */ public double std_dev(double[] A) {... }

13. Chris Murphy – Columbia University13 Extensions to JML Array/Collection Functions  Shuffle/permute the order of the elements  Multiply each element by a constant  Add a constant to each element  Include a new element  Exclude an element Addressing non-determinism  Check in range of possible values  Check in a set of possible values

14. Chris Murphy – Columbia University14 Model Caller Function Caller Function Input(s) Callee Function Callee Function Output Check Metamorphic Properties Check Metamorphic Properties Output Input(s)

15. Chris Murphy – Columbia University15 Implementation /*@ @meta average(\multiply(A, 2)) == \result * 2 */ public double average(double[] A) {... } protected boolean __metaTest_average (double[] A, double result) { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), result * 2); } @ensures __metaTest_average(A, \result);

16. Chris Murphy – Columbia University16 Case Studies We investigated the WEKA and RapidMiner toolkits for Machine Learning in Java For WEKA, we tested four apps:  Naïve Bayes, Support Vector Machines (SVM), C4.5 Decision Tree, and k-Nearest Neighbors For RapidMiner, we tested one app:  Naïve Bayes

17. Chris Murphy – Columbia University17 Experimental Setup For each of the five apps, we specified 4-6 metamorphic properties of selected methods (based on our knowledge of the expected behavior of the overall application) Testing was conducted using data sets from UCI Machine Learning Repository Goal was to determine whether the properties held as expected

18. Chris Murphy – Columbia University18 Findings Discovered defects in WEKA k-NN and WEKA Naïve Bayes related to modifying the machine learning “model”  This was the result of a variable not being updated appropriately Discovered a defect in RapidMiner Naïve Bayes related to determining confidence  There was an error in the calculation

19. Chris Murphy – Columbia University19 Limitations Specifying and checking the properties was simplified by use of the tool, but still requires some knowledge of the app to determine the properties in the first place The approach works well for functions that take input and produce output, but in some case requires more complex properties than can be expressed in JML

20. Chris Murphy – Columbia University20 Future Work Reducing testing time by checking properties in parallel Implementations for other languages More empirical studies to determine effectiveness and applicability to other domains

21. Chris Murphy – Columbia University21 Summary We have presented a testing approach that uses metamorphic testing of individual functions in applications that do not have test oracles These are specified via an extension to the JML specification language We have also presented an implementation framework called Corduroy, which converts the specification of metamorphic properties into test methods

22. Using JML Runtime Assertion Checking to Automate Metamorphic Testing in Applications without Test Oracles Chris Murphy cmurphy@cs.columbia.edu

23. Chris Murphy – Columbia University23 Categories of Metamorphic Properties Additive: Increase (or decrease) numerical values by a constant Multiplicative: Multiply numerical values by a constant Permutative: Randomly permute the order of elements in a set Invertive: Reverse the order of elements in a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set ML apps such as ranking, classification, and anomaly detection exhibit these properties [Murphy SEKE’08]

24. Chris Murphy – Columbia University24 Specifying More Complex Properties /*@ @meta test_myFunc(x, \result) */ public double myFunc (int x) {... } private boolean test_myFunc (int x, double result) {... }

25. Chris Murphy – Columbia University25 Addressing Side Effects // holds result of last call to “average” private double value; /*@ @assignable value; @meta average(\multiply(A, 2)) == value * 2 */ public void average(double[] A) {... value =... // no return value }

26. Chris Murphy – Columbia University26 Addressing Side Effects protected boolean __metaTest_average (double[] A) { double __value = value; // backup try { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), value * 2); } finally { value = __value; // restore } __value