Distributed In Vivo Testing of Software Applications Matt Chu, Christian Murphy, Gail Kaiser Columbia University

2 Chris Murphy, Columbia University Introduction It is infeasible to fully test a large system prior to deployment considering:  different runtime environments  different configuration options  different patterns of usage Extending testing into the deployment environment can help reveal faults

3 Chris Murphy, Columbia University In Vivo Testing We have developed an approach called in vivo testing  Unit tests from the development environment (in vitro) are executed at run- time in the deployment environment (in vivo)  Done without affecting the state of the application However, this approach can incur a large performance overhead

4 Chris Murphy, Columbia University Distributed In Vivo Testing In this work we present a distributed approach that takes advantage of networked “application communities” Members of the community each run fewer tests (thus less overhead) and results are aggregated centrally

5 Chris Murphy, Columbia University Benefits of Distributed In Vivo Testing Amortizing the workload over many instances tackles the high performance cost without sacrificing the quantity of tests being conducted globally Having a community of applications collaboratively working together increases the possibility that an instance will find these erroneous permutations of state

6 Chris Murphy, Columbia University Overview Related work In vivo testing & the Invite framework The distributed Invite framework Experimental results Conclusions and future work

7 Chris Murphy, Columbia University Related Work Perpetual testing [Clarke SAS’00] Skoll [Memon ICSE’04] Gamma [Orso ISSTA’02] CBI [Liblit PLDI’03] MORABIT [Merdes SEM’06]

8 Chris Murphy, Columbia University In Vivo Testing Unit tests are executed in the deployment environment, in the context of the running application, as opposed to a controlled development environment Tests are run continuously as the application runs, at selected points in the program execution Tests are run in a separate process (which has been created as an exact copy of the original), so as not to affect the state of the original process

9 Chris Murphy, Columbia University In Vivo Testing: Defects Targeted 1. Unit tests make incorrect assumptions about the state of objects in the application 2. Possible field configurations were not tested in the lab 3. A legal user action puts the system in an unexpected state 4. An sequence of unanticipated user actions breaks the system 5. Defects that only appear intermittently

10 Chris Murphy, Columbia University Invite: The in vivo testing framework Implemented in Java Uses AspectJ to “weave in” the instrumentation at compile-time Configuration performed by software developer  Frequency of test execution (per method)  Reaction policy Software deployed as usual, with no requirements on end user

11 Chris Murphy, Columbia University Invite: Running a test Run a test?... Method foo is about to be executed Run the test in a separate process Execute testFoo Execute foo no yes Record success/failure

12 Chris Murphy, Columbia University Distributed in vivo testing approach Amortize the cost of test execution by distributing the workload over a number of instances Cover a variety of different platforms, usage patterns, etc. Client/Server architecture Server can distribute tests according to targeting or sampling policies

13 Chris Murphy, Columbia University Distributed Invite framework Invite Server

14 Chris Murphy, Columbia University Experiment Setup Goal: ensure that evenly distributing testing load improves performance by decreasing the number of tests each individual member in the community has to run. Subject: Jetty WebServer 5000 dynamic & static web pages requested from each instance

15 Chris Murphy, Columbia University Experimental Results community size % overhead

16 Chris Murphy, Columbia University Summary Overhead declines as the number of members in the community grows Analytical results:  18 members: 25%  45 members: 10%  90 members: 5% Optimal minimum overhead (from instrumentation) was 2.4%

17 Chris Murphy, Columbia University Limitations & Future Work Ensure that the test does not affect the external system state Improve distribution of tests Adjust frequency of test execution based on context or resource availability Include support for capture/replay, fault localization, etc.

18 Chris Murphy, Columbia University Conclusion Distributed in vivo testing addresses one of the biggest problems of the in vivo testing approach (extremely high performance penalties) by distributing the number of tests that need to be run onto the entire community of applications The approach maintains the same number of tests globally, and also encompasses more diverse environments

Distributed In Vivo Testing of Software Applications Matt Chu, Christian Murphy, Gail Kaiser Columbia University