1. Using Runtime Testing to Detect Defects in Applications without Test Oracles Chris Murphy Columbia University November 10, 2008

2. Chris Murphy – Columbia University2 3 rd -year PhD student at Columbia University Advised by Prof. Gail Kaiser Research interests:  Software Testing  Computer Science Education  Computer-Supported Cooperative Work About Me

3. Chris Murphy – Columbia University3 Introduction This thesis addresses the problem of testing complex, highly configurable systems, particularly those without “test oracles” that indicate what the correct output should be for arbitrary input We adapt a technique that leverages built-in “pseudo-oracles” and perform testing in the deployment environment in order to address limitations regarding defects that reveal themselves only in certain states or with certain input data

4. Chris Murphy – Columbia University4 Overview Problem Statement & Requirements Approach & Hypotheses Model & Architecture Feasibility & Preliminary Results Related Work Expected Contributions

5. Chris Murphy – Columbia University5 Problem Statement Some applications, such as in Machine Learning, do not have test oracles for the general case Even if certain defects may be detectable, others can only be revealed as a result of particular input values, configurations, application states, or runtime environments that may not have been tested prior to deploying a software product

6. Chris Murphy – Columbia University6 Observation Even when there is no oracle in the general case, there can still be a limited subset of inputs such that:  they can at least reveal certain types of defects, e.g. catastrophic failures (crashes), and/or  the expected output can actually be known These inputs may be generated based on past inputs and their respective outputs, as in “Metamorphic Testing” [Chen ’98]

7. Chris Murphy – Columbia University7 Metamorphic Testing Originally designed as an approach for creating follow-up test cases based on those that have not revealed any defects If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce t(x) We can then predict the expected value of f(t(x)) based on the known value of f(x) If f(t(x)) is not as expected, then a defect exists

8. Chris Murphy – Columbia University8 Metamorphic Testing Example Anomaly-based network intrusion detection systems build a “model” of normal behavior In some cases, the model may consider the byte distribution of data in the incoming network packet; anything deemed anomalous causes an alert We cannot know a priori whether a particular packet should cause an alert However, if we permute the order of the bytes, the result (anomalous or not) should be the same as for the original packet, since the system only considers the distribution of the bytes

9. Chris Murphy – Columbia University9 Proposed Approach To address the problem of functions and/or applications that have no test oracle, we use Metamorphic Testing In such applications, to reveal defects that are dependent on input data, configuration, application state, or the runtime environment, we continue Metamorphic Testing in the field, after deployment and during actual usage

10. Chris Murphy – Columbia University10 Approach Details Initial input/output pairs are taken from actual executions We cannot know whether the output is correct but we at least know that the input is something that comes up in practice, and is useful as a valid test case We then apply “metamorphic properties” to get test input, so that we should be able to predict the test output Although we cannot know whether the test output is correct either, if it is not as predicted then there is a defect Since this runs in the field, we have to ensure that users don't notice this testing, e.g. see the test output, experience a sudden performance lag, etc.

11. Chris Murphy – Columbia University11 Hypotheses 1. For programs that do not have a test oracle, conducting Metamorphic Testing within the context of the application running in the field can reveal defects that would not ordinarily otherwise be found 2. This can be done without affecting the application state from the users’ perspective, and with minimal performance overhead

12. Chris Murphy – Columbia University12 1. Automated Metamorphic System Testing: Conducts system-level Metamorphic Testing in the deployment environment 2. Metamorphic Runtime Checking: A separate testing technique that, for individual units (functions), supports the execution of Metamorphic Tests that are executed “from within” the context of the running application Proposed Model

13. Chris Murphy – Columbia University13 Automated Metamorphic System Testing Checks that the metamorphic properties of the entire system hold after execution Treats the application as a black box Multiple invocations run in parallel, and results are compared upon completion User only sees output from the “original” invocation

14. Chris Murphy – Columbia University14

15. Chris Murphy – Columbia University15 Amsterdam: Automated Metamorphic System Testing Framework Metamorphic properties are specified in XML  Input transformation  Runtime options  Output comparison Framework provides out-of-box support for numerous transformation and comparison functions but is extendable to support custom operations Additional invocations are executed in parallel in separate sandboxes that have their own virtual execution environment

16. Chris Murphy – Columbia University16 Metamorphic Runtime Checking For individual units (functions), we check whether the metamorphic properties hold as the application is running, using actual input from real executions and the application’s current state Function arguments are modified according to the metamorphic properties Function is called again (in an isolated sandbox) with the new input Outputs are compared

17. Chris Murphy – Columbia University17 Create a sandbox for the test Run a pre-test? Function foo is about to be executed Execute preTestFoo Execute foo no yes Record success/failure Run a post-test? Execute postTestFoo yes no Create a sandbox for the test Record success/failure Program continues

18. Chris Murphy – Columbia University18 Foundation: In Vivo Testing To facilitate testing “from within” a running program, we will extend In Vivo Testing [Chu ICST’08] In Vivo Tests are analogous to unit tests but they test from within the context of the running application as it executes in the deployment environment, as opposed to a clean slate They test that sequences of actions produce the expected results, no matter what the configuration, state, or runtime environment

19. Chris Murphy – Columbia University19 Complementary Testing Approaches Metamorphic Testing addresses a limitation of In Vivo Testing:  The need for a test oracle In Vivo Testing addresses some limitations of Metamorphic Testing:  Availability of initial test data  Detecting defects that only appear in certain states, configurations, or environments; or occur only intermittently

20. Chris Murphy – Columbia University20 Columbus: Metamorphic Runtime Checking Framework Tests (specifications of metamorphic properties) are written by developers, then select components of the application are instrumented with tests at compile-time Configuration includes: probability of running a test for each function; maximum number of concurrent tests; action to take when test fails; whether to assign tests to separate processor/core Test sandbox can be created by simple “fork” or by creating a virtual execution environment

21. Chris Murphy – Columbia University21 Preliminary Results Identified categories of metamorphic properties in the domain of Machine Learning Detected defects with Metamorphic System Testing Detected defects with In Vivo Testing Detected defects with Metamorphic Runtime Checking

22. Chris Murphy – Columbia University22 Categories of Metamorphic Properties [Murphy SEKE’08] Additive: Increase (or decrease) numerical values by a constant Multiplicative: Multiply numerical values by a constant Permutative: Randomly permute the order of elements in a set Invertive: Reverse the order of elements in a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set ML apps such as ranking, classification, and anomaly detection exhibit these properties

23. Chris Murphy – Columbia University23 Feasibility: Metamorphic System Testing We performed system-level metamorphic testing on various types of Machine Learning applications [Murphy SEKE’08] Detected previously-unknown defects in a real-world network intrusion detection system However, this testing was not automated: inputs were modified with one-off scripts and outputs were compared manually

24. Chris Murphy – Columbia University24 Feasibility: In Vivo Testing We have previously developed an implementation of the In Vivo Testing framework for Java applications called Invite [Chu ICST’08] Targeted towards applications in which defects were not obvious to the user (not necessarily those without test oracles) Detected known defects in OSCache that were not found by traditional unit tests Uses “fork” to create new processes for sandbox

25. Chris Murphy – Columbia University25 We have developed a system by which functions’ metamorphic properties are specified using an extension to JML Specifications converted into metamorphic unit tests by a tool called Corduroy [Murphy ’08]  tests run using JML Runtime Assertion Checking Detected defects in WEKA and RapidMiner machine learning toolkits Feasibility: Metamorphic Runtime Checking

26. Chris Murphy – Columbia University26 Related Work: Absence of Oracles Pseudo-oracles [Davis ACM’81] Testing non-testable programs [Weyuker TCJ’82] Overview of approaches [Baresi ’01]  Embedded assertion languages  Extrinsic interface contracts  Pure specification languages  Trace checking & log file analysis Using metamorphic testing [Chen JIST’02]

27. Chris Murphy – Columbia University27 Related Work: Testing in the Field Perpetual Testing [Osterweil QW’96] Gamma [Orso ISSTA’02] Skoll [Memon ICSE’04] Cooperative Bug Isolation [Liblit RAMSS’04] Failure-Oblivious Computing [Rinard OSDI’04] Security systems that monitor for errors

28. Chris Murphy – Columbia University28 Methodology (1) To further demonstrate feasibility, we will conduct Automated Metamorphic System Testing and Metamorphic Runtime Checking on real-world Machine Learning applications as they run under normal operation in the field We will also show that certain defects would not ordinarily have been detected by using Metamorphic Testing (or other techniques) prior to deployment

29. Chris Murphy – Columbia University29 Methodology (2) To show that our testing approach advances the state of the art in testing applications that have no test oracle, we will compare it to other techniques that could be used to address this same problem  Symbolic execution  Model checking  Program invariants  Formal specification languages

30. Chris Murphy – Columbia University30 Expected Contributions 1. Automated Metamorphic System Testing and a testing framework called Amsterdam 2. Metamorphic Runtime Checking and a testing framework called Columbus 3. A set of guidelines for assisting the formulation and specification of metamorphic properties

31. Chris Murphy – Columbia University31 Using Runtime Testing to Detect Defects in Applications without Test Oracles Chris Murphy cmurphy@cs.columbia.edu