Properties of Machine Learning Applications for Use in Metamorphic Testing Chris Murphy, Gail Kaiser, Lifeng Hu, Leon Wu Columbia University

Introduction We are investigating the quality assurance of Machine Learning (ML) applications Machine Learning applications fall into a class for which it can be said that there is “no reliable test oracle”

Introduction Previously we have investigated approaches to testing such applications by considering properties of their data sets and by using random testing In this work, we seek to adapt Metamorphic Testing [Chen ’98] to these applications and consider their Metamorphic Properties

Contribution Our contribution is a set of Metamorphic Properties that can be used to define these relationships so that Metamorphic Testing can be used as a general approach to testing machine learning applications

Overview Background Testing Approach Findings and Results Future Work and Conclusion

Metamorphic Testing General technique for creating follow-up test cases based on existing ones, particularly those that have not revealed any failure  [Chen ’98, Gotleib COMPSAC’03, Chen STEP’04, Zhou ISFST’04] Use a function’s Metamorphic Properties to predict the output for a particular input, given the known output for another input  For example, if we know sin(x)=y, then we know: sin(x+2 π ) = y and sin(-x) = -y

Related Work Applying metamorphic testing to situations in which there is no test oracle [Chen IST’02] There has been much research into applying Machine Learning techniques to software testing, but not much the other way around Testing of intrusion detection systems has typically addressed quantitative measurements but does not seek to ensure that the implementation is free of defects

Machine Learning Fundamentals Data sets consist of a number of examples, each of which has attributes and a label In the first phase (“training”), a model is generated that attempts to generalize how attributes relate to the label (if they exist) In the second phase, the model is applied to a previously-unseen data set with unknown labels to produce a classification (or, in some cases, a ranking)

Sample Data Set For supervised machine learning 27,81,88,59,42,16,88, 0 82, 6,51,47, 5, 4, 1, 0 22,72,11,84,96,24,44, 1 4,77,91,86,89,77,61, 1 76,11, 4,51,43, 2,79, 0 6,33,44,18,52,63,94, 0 77,36,91,81,47, 3,85, 1 39,17,15, 2,90,70,13, 0 8,58,42,41,74,87,68, 1 examples attributes labels

Applications Investigated MartiRank  Specifically designed for potential future experimental use in predicting impending electrical device failures by ranking them according to likelihood of failure  Seeks to find the combination of segmenting and sorting the data that produces the best result Support Vector Machines (SVM)  Seeks to find a hyperplane that separates examples from different classes  SVM-Light has a ranking mode based on the distance from the hyperplane PAYL  Anomaly-based intrusion detection system (IDS)  Builds a model of “normal” network traffic based on byte distribution, and reports any anomalies

Approach Previously tested such applications by analysis of the data sets and algorithms, and by using equivalence partitions to guide random testing In this work, we use our knowledge of MartiRank to devise a set of Metamorphic Properties, and then see if they also apply to SVM and PAYL We then use these properties to guide testing of these applications

MartiRank Metamorphic Properties Additive  If each value in the data set is increased by a constant, the final ranking should be unchanged Multiplicative  If each value in the data set is multiplied by a positive constant, the final ranking should be unchanged Permutative  If the order of the data is permuted, the final ranking should be unchanged (assuming distinct values in the data set)

MartiRank Metamorphic Properties Invertive  If each value in the data set is multiplied by a negative constant, the final ranking should be in the reverse order Inclusive  In the “testing phase”, if the model is already known, it should be possible to create an example in the testing data such that it is guaranteed to be at the top of the ranking Exclusive  If an example is removed from the testing data, the final ranking should be unchanged

Testing MartiRank Its invertive property should hold for the labels in the training data, too  Multiplying the labels by –1 should yield a model that, when applied to the same testing data, will result in the reverse ordering Negative labels were not considered by the developer and a defect was revealed through Metamorphic Testing

Applying Approach to SVM SVM exhibits all six Metamorphic Properties A defect was found in SVM-Light by using its permutative property  Permuting the input data led to different models (and then different rankings)  Caused by “chunking” data for use by an approximating variant of optimization algorithm

Applying Approach to PAYL PAYL exhibits all six Metamorphic Properties  Even though it is unsupervised ML Two defects were found by using its exclusive property  Removing a value from the training data did not cause it to be considered anomalous later on  It also caused other values to be considered anomalous

Future Work and Conclusion We have identified six Metamorphic Properties that we believe exist in many machine learning applications:  additive, multiplicative, permutative, invertive, inclusive, and exclusive These properties were used to find new defects in the ML applications of interest Further investigation could involve applying these properties to other, larger ML applications, and looking to classify other properties

Properties of Machine Learning Applications for Use in Metamorphic Testing Leon Wu Columbia University