1. Applications of Metamorphic Testing Chris Murphy University of Pennsylvania cdmurphy@cis.upenn.edu November 17, 2011

2. About Me Lecturer, University of Pennsylvania PhD-Computer Science, Columbia Univ, 2010 Advisor: Prof. Gail Kaiser Research: software testing, CS education Seven years experience in software development industry

3. Problem: When testing a piece of software, how can we know that we’ve created enough test cases?

4. Problem: When testing a piece of software, how can we create more test cases? Solution: We use properties of the software to create new test cases from existing ones (particularly those that have not failed). Result: This approach, known as metamorphic testing, is more effective at testing certain types of software than other approaches.

5. Today's Talk What is metamorphic testing? How is metamorphic testing used to find bugs in software? How can metamorphic testing be applied to applications that do not have test oracles? What are the open research questions related to metamorphic testing?

6. UPenn CIS 573 Software Engineering Graduate-level software engineering course Just over 100 students Focuses on software maintenance issues: – Testing – Formal verification – Debugging – Fault Localization – Refactoring

7. It's your first day of work as a software engineer at BloobleSoft. Your boss gives you 6,000 lines of code and a specification and says “find the bugs”. Where would you start?

8. Specification Code Test Case Generation Strategy Test Cases Test Cases Test Cases Test Cases

9. You've created 4,837 test cases. All of them pass. How do you know when you're done creating test cases?

10. Testing Requirements Test Cases Desired Adequacy Level Measurable Adequacy Criteria Acceptable? Coverage Level

11. Your test cases are achieving 100% coverage. None of them have found any bugs. Are those test cases useful?

12. Maybe those test cases can be used to create new test cases. The more test cases, the better. Right? This is the idea behind “metamorphic testing”. [Chen et al., HKUST TR CS-98-01, 1998]

13. A (really, really, really, really) simple example

14. Let's say you're testing a cosine function. (I know, I know...) How could we use this test case to create new test cases? You have a test case {45º, 0.7071}, i.e. cos(45º) = 0.7071

15. We know that the cosine function exhibits certain properties. That is, if we make certain changes to the input, we can predict the effect on the output. These are referred to as “metamorphic properties”.

16. What are the metamorphic properties of the cosine function? cos(x + 360º) = cos(x) cos(x + 180º) = -1 * cos(x) cos(x - 360º) = cos(x) That is, if we add 360 to the input, the output should not change.

17. Given our original test case {45°, 0.7071}, we can create three follow-on test cases. Property: cos(x + 360º) = cos(x) Input: 45º + 360º = 405º Output: cos(45º) = 0.7071 Property: cos(x - 360º) = cos(x) Input: 45º - 360º = -315º Output: cos(45º) = 0.7071 Property: cos(-1 * x) = -1 * cos(x) Input: -1 * 45º = -45º Output: -1 * cos(45º) = -0.7071

18. x f f(x) Initial test case {x, f(x)} t(x) f f(t(x)) Follow-on test case {t(x), f(t(x))} t A metamorphic property of a function f is a pair of functions (t, g) such that f(t(x)) = g(f(x)) for all inputs x g(f(x)) g =

19. but wait, isn’t that the same as…

20. Program invariants: -1 ≤ cos(x) ≤ 1 Algebraic properties: cos²(x) = 1 – sin²(x) Describe legal ranges/values of a function, but not how it should react when the input is changed. Describe the relationships between multiple functions, but not a single function.

21. simple categories of properties

22. abcdef Initial test case cebafd Permute 2a2b2c2d2e2f Multiply s sum s Add a+2b+2c+2d+2e+2f+2 s+12 2s sum s+g abcdef Include gabcde Exclude s-f sum

23. abcdef Initial test case #1 s sum ghijkl Initial test case #2 t sum Compose s+t sum abcdef ghijkl Combination of properties 2s+2t sum 2h2d2a2k2e2g 2i2c2l2f2b2j

24. Common Metamorphic Properties Additive: Increase (or decrease) numerical values by a constant Multiplicative: Multiply numerical values by a constant Permutative: Randomly permute the order of elements in a set Invertive: Create the “opposite” of a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set Compositional: Compose a set [Murphy et al., SEKE’08]

25. Other Types of Properties Noise-based: include input values that will not affect the output Semantically Equivalent: create inputs that are have the same “meaning” as the original Heuristic: create inputs that are “close” to the original Statistical: create inputs that exhibit the same statistical properties

26. one more example

27. Consider a function that takes a set of Points (x-y coordinates) and calculates the total distance from the first to the last, via the rest. What are that function’s metamorphic properties?

28. Okay, I think I get it. But does it really work?!?!

29. In order to find bugs… But how can this be?!?! 1. The original test case must pass, even though there is a bug. 2. The follow-on test case must fail.

30. /* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; }

31. /* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1}PASS! 100% statement coverage! 100% branch coverage!

32. /* Return the smallest value in the array */ int findMin(int A[]) { int min = A[0]; for (int i = 1; i < A.length-1; i++) { if (A[i] < min) min = A[i]; } return min; } Test case { {2, 1, 4, 3}, 1} Metamorphic property: If we permute the input, the output remains the same. Follow-on test case: { {4, 2, 3, 1}, 1}FAIL!

33. metamorphic testing in the real world

34. Bioinformatics [Chen et al., BMC Bioinf., 2009] Machine Learning [Xie et al., JSS, 2011] Network Simulation [Chen et al., FTDS, 2009] Computer Graphics [Guderlei et al., QSIC, 2007]

35. what types of applications is metamorphic testing good for?

36. Applications that deal primarily with numerical input and numerical output. Applications that do not have test oracles. Applications that use graph-based algorithms. Compilers. [Zhou et al., ISFST’04]

37. Specification Program Test Input Actual Output Expected Output Oracle

38. what if there is no oracle?

39. Machine Learning

40. Discrete Event Simulation

41. x f f(x) t(x) f f(t(x)) Actual t If f(t(x)) = g(f(x)) that does not mean that the output is correct. g =? But if f(t(x)) != g(f(x)) then one (or both) must be incorrect. g(f(x)) Expected

42. example: RapidMiner

43. RapidMiner is a suite of machine learning algorithms implemented in Java. That is: c = Classify(M(T), e) In its NaïveBayes implementation, a confidence level c is reported whenever it classifies an example e using a model M created from a training data set T.

44. We expect that if we modify T to include an extra instance of e, then the confidence level should double, since we are twice as certain about the classification. That is: Classify(M(T+e), e) = 2 * Classify(M(T), e) Our testing detected violations of this property, thus revealing a bug. [Murphy et al., ICST’09]

45. empirical study

46. Goal: Show that metamorphic testing is more effective than other techniques at finding bugs in applications without test oracles. Approach: Use mutation analysis to insert faults into the applications, and see how many are detected using various techniques.

47. Application domains investigated: 1. Machine Learning (C4.5, MartiRank, Support Vector Machines, PAYL) 2. Discrete Event Simulation (JSim) 3. Information Retrieval (Lucene) 4. Optimization (gaffitter)

48. Techniques investigated: 1. Metamorphic Testing 2. Runtime Assertion Checking 3. Partial Oracle

49. Experimental Results [Murphy et al., ISSTA’09]

50. can we do better?

51. That experiment used application- level metamorphic properties. What if we test at the function level, too? This is known as Metamorphic Runtime Checking. [Murphy et al., TR CUCS-042-09, 2009] And continuously conduct those tests while the software is running?

52. Experimental Results

53. research directions

54. When I run my test, I see that the metamorphic property is violated. Does that mean there's a bug? How can we know whether the metamorphic properties are sound? Well, not necessarily....

55. I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? How can we know whether the set of metamorphic properties is complete? Well, not necessarily....

56. I've used the guidelines to identify as many metamorphic properties as I could. Does that mean that's all of them? Could we detect (likely) metamorphic properties automatically? Well, not necessarily....

57. I have a function for which I expect that, if I double the input, the output should be doubled. Could I verify that property without actually executing the code? Can metamorphic properties be verified statically? Well, probably....

58. summary

59. Metamorphic testing is a method of creating new test cases from existing ones. It depends heavily on the software’s metamorphic properties, which are often numerical. Metamorphic testing is particularly effective at finding bugs in applications that do not have test oracles.

60. thanks!

61. Applications of Metamorphic Testing Chris Murphy University of Pennsylvania cdmurphy@cis.upenn.edu