Download presentation
Presentation is loading. Please wait.
1
Efficient Systematic Testing for Dynamically Updatable Software Christopher M. Hayden, Eric A. Hardisty, Michael Hicks, Jeffrey S. Foster University of Maryland, College Park
2
Dynamic Software Updating (DSU) Performing updates to software at runtime has clear benefits: Increased software availability No need to terminate active connections / computation … but can we trust updated software? Critical to ensure updates are safe 2
3
Our Contributions Verification of DSU through testing: Testing Procedure Test Minimization Algorithm Experiments Effectiveness of Minimization Empirical study of Update Safety / Safety Checks 3
4
DSU Safety DSU creates the opportunity for new sources of bugs: Faulty state transformation Unsafe update timing Safety Checks – restrict when updates may be applied Activeness Safety / Con-freeness Safety 4
5
Activeness Safety (AS) AS prevents updates to active code In this example, no patch updating main or foo is allowed: 5 main() { foo(); … baz(); } foo() { … bar(); }
6
Con-freeness Safety (CFS) CFS (Stoyle, et al ‘05) allows updates to active code only when type safety can be ensured In this example, no patch updating the signature of baz or bar is allowed: 6 main() { foo(); … baz(); } foo() { … bar(); }
7
DSU Testing Safety Checks offer limited guarantees: CFS and AS ensure type-safe execution AS ensure that you never return to old code following an update Neither of these properties ensure safe update timing We propose testing to verify the correctness of allowed update points: Use existing suite of application system tests Ensure that updating anywhere during the execution of those tests results in an execution that passes the test. 7
8
Testing Procedure Approach: Instrument application to trace update points Execute system test and gather initial trace For each update point in the initial trace, perform an update test: force an update at that point while executing the system test Potential Update Points Trace Start 8
9
Testing Procedure Approach: Instrument application to trace update points Execute system test and gather initial trace For each update point in the initial trace, perform an update test: force an update at that point while executing the system test ✔ initial trace 9
10
Testing Procedure Approach: Instrument application to trace update points Execute system test and gather initial trace For each update point in the initial trace, perform an update test: force an update at that point while executing the system test ✔ initial traceupdate tests ✔ ✘ ✔ 10
11
Update Test Minimization Program traces may have thousands or millions of update points Many update tests have the same behavior for a given patch we can eliminate redundant tests baz() {…} Patch A void main() { foo(); bar(); baz(); } Version 0 foo() {…} bar() {…} baz() {…} Patch B All update points yield same behavior All update points yield distinct behavior 11
12
Minimization Algorithm Execution events are traced if they have the potential to conflict with a patch A event conflicts with a patch if applying before the event might produce a different result than applying after the event Example: function calls, global variable accesses Trace the execution of a test T on P 0 Iterate through the trace noting the last update point each time we reach a conflicting trace element Run only the identified update tests T n 12
13
Experimental Results 13
14
Experimental Setup Based testing infrastructure on top of the Ginseng DSU system (Neamtiu, et al): Modified to support tracing and updating at pre- selected update points Insertion of explicit update points before each function call to approximate more liberal systems Disabled safety checking (CFS) for experiments Tested 3 years of patches to OpenSSH and vsftpd (only report OpenSSH in this talk) 14
15
Program Modifications 15 foo() { while (1) { // main loop update(); extract {... // main loop body } extract {... // after main Loop } Identify Long-running loops Add a Manually Selected Update Point Perform Loop Body Extraction Perform Continuation Extraction
16
Experiments: Update Test Suite How many update tests must be run to test real- world updates to real-world applications? How effective is minimization at eliminating redundant tests? 16
17
Update Test Suite Size: OpenSSH # to next version Reduction SigFunTyp e All PointsActiveness-Safe Points 03985580,871 31,791(95%)35,314 3,027(91%) 1060705,322 1,795(~100%)587,578 1,717(~100%) 2523811638,720 63,011(90%)20,902 2,353(89%) 30180772,198 4,324(99%)638,803 3,775(99%) 41317210773,086 27,399(96%)21,343 1,564(93%) 50241878,235 17,398(98%)111,950 1,723(98%) 6625710879,668 47,092(95%)44,278 2,139(95%) 7417912918,717 89,601(90%)100,854 4,141(96%) 80723973,364 34,293(96%)61,724 2,070(97%) 9101577933,514 52,356(94%)61,051 2,891(95%) Total8,053,695 369,060(95%)1,683,797 25,400(98%) 17
18
Empirical Study of Update Safety How many failures occur when applying updates arbitrarily? How many failures occur when applying updates subject only to the AS and CFS safety checks? 18
19
Safety: OpenSSH 19 to next version All PointsCFS PointsAS Points UpdateSigFunTypeFailedTotalFailedTotalFailedTotal 0398519,715580,871068,044035,314 10600705,3220 0587,578 2523811306,965683,7201,68875,307420,902 301800772,1980 0638,803 41317210565,681773,086609110,63338021,343 5024110,703878,2350130,0000111,950 6625710163,333879,66844,46 1 96,18311044,278 741791211,380918,717180,0701100,854 807233973,3640261,885061,724 9101577357,919933,51424121,337061,051 Total1,435,69 9 8,053,69 5 46,78 3 2,420,97 9 4951,683,79 7
20
void handle_upload_common() { ret = do_file_recv(); } void do_file_recv() { … // receive file if (ret == SUCCESS) write(226, “OK.”); return ret; } Version 0 void handle_upload_common() { ret = do_file_recv(); if (ret == SUCCESS) write(226, “OK.”); } void do_file_recv () { … // receive file return ret; } Version 1 (patch) Unsafe Timing: Version Inconsistency (vsftpd) 20
21
Unsafe Timing: Version Inconsistency void foo() { bar(); … baz(); } void bar() { … } void baz() { dig(); … } Version 0 void foo() { bar(); … baz(); } void bar() { dig(); … } void baz() { … } Version 1 (patch)
22
Manually Selected Update Points 22 to next version Safety #TestsSigFunTypeReductionFailedTotal 0753985566 (0%)0566 175060630 592(6%)0630 276523811568 (0%)0568 3910180783 770(2%)0783 4911317210782 (0%)0782 51040241860 841(2%)0860 6104625710859 (0%)0859 7104417912850 (0%)0850 81050723868 823(5%)0868 9104101577833 (0%)0833 Tota l 7,59 9 7,48 4 (2% ) 07,59 9
23
Summary We have argued that verification is necessary to prevent unsafe updates Provided empirical evidence that AS/CFS cannot prevent all unsafe updates We have presented an approach for testing dynamic updates We have presented and evaluated a minimization strategy to make update testing more practical 23
24
Discussion Questions Given that AS cannot ensure correctness (both in theory and in practice), should DSU implementations continue to rely on it? What standards for verification should be required of DSU system benchmarks? Are there other assumptions of DSU that are appropriate for empirical evaluation? 24
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.