School of Electrical Engineering and Computer Science University of Central Florida Anomaly-Based Bug Prediction, Isolation, and Validation: An Automated Approach for Software Debugging Martin Dimitrov Huiyang Zhou

Background: Terminology Defect : Erroneous piece of code (a bug) I nfection : The defect is triggered => the program state differs from what the programmer intended. Failure : An observable error (crash, hang, wrong results) in program behavior. University of Central Florida Terminologies are based on the book “Why Programs Fail” by Andreas Zeller. 1

Background: From Defects to Failures University of Central Florida Program execution Variable and input values Sane state Erroneous code Infected state Observer sees failure Figure from the book “Why Programs Fail” by A. Zeller 2

Motivation The typical process of software debugging involves: 1)Examine the point of program failure and reason backwards about the possible causes. 2)Create a hypothesis of what could be the root cause. 3)Modify the program to verify the hypothesis. 4)If the failure is still there, the search resumes. Software debugging is tedious and time consuming ! In this work we propose an approach to automate the debugging effort and pinpoint the failure root cause. University of Central Florida 3

Presentation Outline Motivation Proposed approach Detecting anomalies (step 1) Isolating relevant anomalies (step 2) Validating anomalies (step 3) Experimental methodology Experimental results Conclusions University of Central Florida 4

Proposed Approach University of Central Florida mov... cmp... jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction 5

Proposed Approach University of Central Florida cmp... jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... 5

Proposed Approach University of Central Florida jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... jl... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... jl... test... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... jl... test... jne... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... jl... test... jne... mov... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida inc... cmp... jl... test... jne... mov... call... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... 5

Proposed Approach University of Central Florida A program failure is observed: –Crash –Hang –Incorrect results, etc. Start the automated debugging process The output of our approach is a ranked list of instructions (the possible root-cause of failure) inc... cmp... jl... test... jne... mov... call... mov... movl... inc... cmp... Dynamic Instruction Stream time Failure point jr... movl... Failure 5

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida mov... cmp... jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida cmp... jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida jge... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... mov... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... lea... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... test... movl... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... test... jne... inc... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... test... jne... mov... cmp... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... test... jne... mov... call... jl... movl... inc... cmp... Dynamic Instruction Stream time Current instruction jr... movl... anomaly 6

Proposed Approach Step 1: Detect anomalies in program execution University of Central Florida inc... cmp... jl... test... jne... mov... call... mov... movl... inc... cmp... Dynamic Instruction Stream time Failure point jr... movl... Failure anomaly Each anomaly constitutes a hypothesis for the root cause of program failure. 6

Proposed Approach Step 2: Isolate the relevant anomalies University of Central Florida inc... cmp... jl... test... jne... mov... call... mov... movl... inc... cmp... Dynamic Instruction Stream time Failure point jr... movl... Failure anomaly Create dynamic forward slices from the anomalies to the failure point. Discard anomalies which do not lead to the failure point. 7

Proposed Approach Step 3: Validate the isolated anomalies University of Central Florida inc... cmp... jl... test... jne... mov... call... mov... movl... inc... cmp... Dynamic Instruction Stream time Failure point jr... movl... Failure anomaly Automatically “fix” the anomaly and observe if the program still fails. 8

Proposed Approach Step 3: Validate the isolated anomalies University of Central Florida inc... cmp... jl... test... jne... mov... call... mov... movl... inc... cmp... Dynamic Instruction Stream time jr... movl... If the failure disappears we have a high confidence the root cause have been pinpointed. Success No failure 8

Detecting Program Anomalies (Step 1) When infected by a software bug the program is likely to misbehave: –Out-of-bounds addresses and values –Unusual control paths –Page faults –Redundant computations, etc. Anomaly detection: Infer program specifications from passing runs and turn them into ‘soft’ assertions. –Learn program invariants during passing runs (e.g. variable “i” is always between 0 and 100 ) –Flag violated invariants during the failing run (e.g. Report anomaly if variable “i” is 101 ) University of Central Florida 9

Detecting Program Anomalies We use several anomaly detectors to monitor a large spectrum of program invariants and catch more bugs. DIDUCE [S. Hangal et al., ICSE 2002 ] –Instructions tent to produce values/addresses within a certain range (e.g. 0 <= i <= 100). Detect violations of these invariants. AccMon [P. Zhou et al., MICRO ] –Only a few static instructions access a given memory location (load/store set locality). Signal an anomaly if memory access does not belong to the load/store set. LoopCount –Detect abnormal number of loop iterations. University of Central Florida 10

Detecting Program Anomalies void more_arrays () {... a_count += STORE_INCR ; /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection : */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } University of Central Florida Heap Memory 0x80bfe6c 0x80bfe70 0x80bfe74 0x80bfe78 0x80bfe7c 0x80bfe80 data size data size a_count  v_count  11

Detecting Program Anomalies void more_arrays () {... a_count += STORE_INCR ; /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection : */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } University of Central Florida DIDUCE: Address of store instruction is out of normal range. AccMon: store instruction is not in store set of this memory location. LoopCount: Loop iterates more times than usual. DIDUCE: Address of store instruction is out of normal range. (false – positive) AccMon: store instruction is not in store set of this memory location. (false – positive) LoopCount: Loop iterates more times than usual. (false – positive) DIDUCE24 anomalies AccMon68 anomalies LoopCount36 anomalies 12

Detecting Program Anomalies: Architectural Support DIDUCE and AccMon capture invariants using limited size caches structures, as proposed in previous work LoopCount utilizes the existing loop-branch predictor to detect anomalies. Advantages and disadvantages of hardware support: +Performance efficiency +Portability +Efficient ways to change or invalidate dynamic instructions -Limited hardware resource may become a concern University of Central Florida 13

Isolating Relevant Anomalies (Step 2) Anomaly detectors alone are NOT effective for debugging: –May signal too many anomalies / false positives –Tradeoff between bug coverage and number of false positives Our solution: –Allow aggressive anomaly detection for maximum bug coverage –Automatically isolate only the relevant anomalies by constructing dynamic forwards slices from the anomaly to the failure point University of Central Florida 14

Isolating Relevant Anomalies: Architectural Support Add token(s) to each register and memory word. Detected anomalies set a token associated with the destination memory word or register. Tokens propagate based data dependencies. When the program fails, examine the point of failure for token. University of Central Florida 15

Isolating Relevant Anomalies: Architectural Support University of Central Florida void more_arrays () {... /* Copy the old arrays. */ for (indx = 1; indx < old_count; indx++) arrays[indx] = old_ary[indx]; /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ }... MemoryToken 0x80bfe7c 0x80bfe80 0x80bfe84 0x80bfe88 0x80bfdf8 0x80bfdfc 0x80bfe00 0x80bfe04 0x80bfe08 Failure mov %ebx,0xc(%edx) 16

Isolating Relevant Anomalies: Architectural Support Problem: Many tokens for each memory location/ register Solution: –We leverage tagged architectures for information flow tracking. –Use only one token (1 bit) (i.e., shared by all anomalies ) –We leverage delta debugging [A. Zeller, FSE 1999 ] to isolate the relevant anomalies automatically. University of Central Florida DIDUCE24 anomalies AccMon68 anomalies LoopCount36 anomalies DIDUCE3 anomalies AccMon2 anomalies LoopCount4 anomalies Number of Initial AnomaliesNumber of Isolated Anomalies 17

Delta-Debugging University of Central Florida 18

Validating Isolated Anomalies (Step 3) Validate the remaining anomalies by applying a “fix” and observing if the program failure disappears. Our “fix” is to nullify the anomalous instruction (turn it into no-op) If the program succeeds, we have a high confidence we have found the root cause (or at least broken the infection chain) University of Central Florida 19

Validating Isolated Anomalies University of Central Florida void more_arrays () {... /* Initialize the new elements. */ for (; indx < v_count; indx++){/* defect */ arrays[indx] = NULL; /* infection */ /* Free the old elements. */ if (old_count != 0){ free (old_ary); /* crash */ } MemoryToken 0x80bfe6c 0x80bfe70 0x80bfe74 0x80bfe78 0x80bfe7c 0x80bfe80 0x0 data size data 0x0 size Success The “size” information is not corrupted and the program terminates successfully. 20

Validating Isolated Anomalies Four possible outcomes of our validation step: University of Central Florida > Segmentation Fault Failure Program fails the same way as before. > Hello World ! Success Program produces expected output. >... (hang) Unknown Program fails in an unexpected manner. > Hello Martin No crash Program does not crash, but produces unexpected results. Rank isolated anomalies based on the outcome: succeed (highest), no crash, unknown, failure (lowest) In our running example the root-cause is ranked #1. 21

Experimental Methodology Implemented a working debugging tool using binary instrumentation (PIN). Evaluated applications from BugBench [S. Lu et al., Bugs 2005] and gcc compiler. University of Central Florida ApplicationLines of Code Defect LocationDefect Description bc ,042storage.c: 176Incorrect bounds checking causes heap buffer overflow gzip ,163gzip.c: 1009Buffer overflow due to misuse of library call strcpy ncompress defects 1,922compress42.c: 886 and 1740 Buffer overflow due to misuse of library call strcpy Incorrect bounds checking causes stack buffer underflow polymorph polymorph.c: 200Incorrect bounds checking causes stack buffer overflow man-1.5h14675man.c:998Incorrect loop exit condition causes stack buffer overflow gcc ,000combine.c: 4013Incorrect call to apply_distributive_law causes a loop in the RTL tree 22

Experimental Results University of Central Florida ApplicationInitial Anomalies Isolated Anomalies Validated (Application Succeeds) Defect rank DALDALDAL bc gzip ncompress (strcpy defect) ncompress (stack underflow) 24n/a polymorph man-1.5h gcc

Case Study: GCC University of Central Florida 24

GCC Defect University of Central Florida 25

GCC Fix University of Central Florida 26

Experimental Results Compared to Failure-Inducing Chops University of Central Florida ApplicationFailure-Inducing Chops Proposed Approach bc gzip ncompress (strcpy defect) 41 ncompress (stack underflow) 111 polymorph man-1.5h1n/a1 gcc

Limitations No failure, no bug detection –Un-triggered bugs –Bugs are triggered but output is correct Target at bugs in sequential programs University of Central Florida 28

Conclusions We present a novel automated approach to pinpoint the root causes of software failures: –Detect anomalies during program execution. –Isolate only the relevant anomalies. –Validate isolated anomalies by nullifying execution results Our experimental results demonstrate that we accurately pinpoint the defect even for large programs such as gcc. University of Central Florida 29

Questions The tool is available for download at: University of Central Florida 30