Download presentation

Presentation is loading. Please wait.

Published byBailee Points Modified about 1 year ago

1
Delta Debugging and Model Checkers for fault localization Amin Alipour Note: Some slides/figures in this presentations has been used/adapted from presentations by Andreas Zeller, Tevfik Bultan, and Alex Groce.

2
Outline Software Fault – some facts Delta debugging – Simplifying test cases – Isolating failure inducing parts in test cases – Search in space Model checking – Background – Distance metrics Conclusion

3
Software faults Software fault/flaw/bug perturbs the state of a program to an error state. Error state can propagates through the execution of the program and cause a failure. Failure is manifestation of error.

4
Software debugging What we have for debugging? – Program – Set of test cases. –…–… For maintainable debugging of failures: – We need to understand the test case/failure. – We need to identify the location of faults. (Fault Localization) Can we automate it?

5
Approaches to Fault Localization Program Slicing Program Spectra Statistical Reasoning Delta Debugging Model Checking

6
Delta Debugging Goal: -Removing components irrelevant to the failure from test cases. – It can improve comprehension of the failure. Delta debugging comes with two techniques: – Simplification (minimization) of test cases, and – Isolation of failure-inducing parts from test cases.

7 Delta Debugging Failing test cases are usually cluttered by unnecessary/irrelevant things. ……. All Windows 3.1 Windows 95 Windows 98 Windows ME Windows2000 Windows NT Mac System 7 Mac System 7.5 Mac System 7.6.1

8
Simplification of test cases Goal: – Minimizing the size of a failing test case, c F. c F = 1 2 ... n Minimizing test cases requires checking all subset of s. Delta debugging simplifies a failing test case c F to a 1-minimal test case. 1-minimal failing test case: – A failing test case is 1-minimal, if any part of it ( i ) is removed, the failure will disappear.

9
Simplification Algorithm i = c F i Test each 1, 2,... n and each 1, 2,..., n There are four possible outcomes 1.Some i causes failure – Partition i to two and continue with i as the test set 2.Some i causes failure – Continue with i as the test set with n 1 subsets 3.No test causes failure – Increase granularity by generating a partition with 2n subsets 4.The granularity can no longer be increased – Done, found the 1-minimal subset

10
Simplification- Example n = 2 n = 4 n = 3 n = 2 n = 4 n = 3 Granularity

11
Simplification Example 2 1 F 2 P 3 P 4 P 5 F 6 F 7 P 8 P 9 P 10 F 11 P 12 P 13 P

12
Simplification Example 2-cont’d 14 P 15 P 16 F 17 F 18 F 19 P 20 P 21 P 22 P 23 P 24 P 25 P 26 F

13
……. All Windows 3.1 Windows 95 Windows 98 Windows ME Windows2000 Windows NT Mac System 7 Mac System 7.5 Mac System 7.6.1

14
Isolation of Failure-inducing part from test case Even in minimal test cases, there are still some elements in the minimal test case that are not directly related to the failure. – E.g., a minimal test case for a C compiler, still needs to have some symbols like: {,}, or variable declarations for the validity of test input that might be irrelevant to the failure. #define SIZE 20 Double mult(double z[], int n) { int i, j; i = 0; for(j=0;j

15
Isolation of Failure-inducing part from a test case How to isolate failure-related parts? – Find a pair of passing and failing input that are very similar and contrast them. #define SIZE 20 Double mult(double z[], int n) { int i, j; i = 0; for(j=0;j

16
Isolation Algorithm Narrow down the gap between passing and failing test case, by removing their differences and making them more similar.

17
Isolation Example 2 F 4 F 7 P 6 P 5 P 3 P 1 P

18
Cause for a failure Can we use the isolation technique to find causes of the failure?

19
Cause for a failure - example

20
cause of a failure - example

21
Cause Transitions rfrf rprp a a a b b c l1l1 l2l2 lili L 1+1 ljlj L j+1 Cause Cause Transition

22
Discussion on delta debugging It scales well. It requires minimal information about the program and its specification. There are several extensions to it: – Hierarchal Delta debugging – Isolating schedules in concurrent systems. – Isolating failure-inducing changes in repositories.

23
Model Checkers for fault localization

24
Model Checking Problem Model Checker Program/Model Specification/ assertions Satisfied Counter-example

25
Fault Localization with Model Checkers Model Checkers can perform different queries on program paths and states. These queries can be used for fault localization: – Contrasting – Distance Metrics – Max-SAT

26
Explanation with Distance Metrics How it’s done: Model checker P+spec First, the program (P) and specification (spec) are sent to the model checker.

27
Explanation with Distance Metrics How it’s done: Model checker P+spec C The model checker finds a counterexample, C.

28
Explanation with Distance Metrics How it’s done: Model checker BMC/constraint generator P+spec C The explanation tool uses P, spec, and C to generate (via Bounded Model Checking) a formula with solutions that are executions of P that are not counterexamples

29
Explanation with Distance Metrics How it’s done: Model checker BMC/constraint generator P+spec C S Constraints are added to this formula for an optimization problem: find a solution that is as similar to C as possible, by the distance metric d. The formula + optimization problem is S

30
Explanation with Distance Metrics How it’s done: Model checker BMC/constraint generator P+spec C Optimization tool S -C An optimization tool (PBS, the Pseudo-Boolean Solver) finds a solution to S: an execution of P that is not a counterexample, and is as similar as possible to C: call this execution -C

31
Explanation with Distance Metrics Model checker BMC/constraint generator P+spec C Optimization tool S -C C ss Report the differences ( s) between C and –C to the user: explanation and fault localization

32
“SSA” Transformation int main () { int x, y; int z = y; if (x > 0) y--; else y++; z++; assert (y == z); } int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); }

33
Transformation to Equations int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); } (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 == z1)

34
Transformation to Equations int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); } (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 == z1) Uninitialized variables in CBMC are unconstrained inputs.

35
Transformation to Equations int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); } (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 == z1) CBMC (1) negates the assertion

36
Transformation to Equations int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); } (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 != z1) (assertion is now negated)

37
Transformation to Equations int main () { int x0, y0; int z0 = y0; y1 = y0 - 1; y2 = y0 + 1; guard1 = x0 > 0; y3 = guard1?y1:y2; z1 = z0 + 1; assert (y3 == z1); } (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 != z1) then (2) translates to SAT and uses a fast solver to find a counterexample

38
Execution Representation (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 != z1) Remove the assertion to get an equation for any execution of the program

39
Execution Representation (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 != z1) Execution represented by assignments to all variables in the equations x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample

40
Execution Representation (z0 == y0 y1 == y0 – 1 y2 == y0 + 1 guard1 == x0 > 0 y3 == guard1?y1:y2 z1 == z0 + 1 y3 == z1) Use the assertion to find a passing trace. x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Passing Trace

41
Execution Representation x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample Execution represented by assignments to all variables in the equations x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Successful execution

42
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample d = number of changes ( s) between two executions x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Successful execution

43
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample d = number of changes ( s) between two executions x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Successful execution

44
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample d = number of changes ( s) between two executions x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Successful execution 1

45
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample d = number of changes ( s) between two executions x0 == 0 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == false y3 == 6 z1 == 6 Successful execution d = 3 3 is the minimum possible distance between the counterexample and a successful execution

46
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample To compute the metric, add a new SAT variable for each potential x0 == (x0 != 1) y0 == (y0 != 5) z0 == (z0 != 5) y1 == (y1 != 4) y2 == (y2 != 6) guard1 == !guard1 y3 == (y3 != 4) z1 == (z1 != 6) New SAT variables

47
The Distance Metric d x0 == 1 y0 == 5 z0 == 5 y1 == 4 y2 == 6 guard1 == true y3 == 4 z1 == 6 Counterexample And minimize the sum of the variables (treated as 0/1 values): a pseudo-Boolean problem x0 == (x0 != 1) y0 == (y0 != 5) z0 == (z0 != 5) y1 == (y1 != 4) y2 == (y2 != 6) guard1 == !guard1 y3 == (y3 != 4) z1 == (z1 != 6) New SAT variables

48
Explanation with Distance Metrics Model checker BMC/constraint generator P+spec C Optimization tool S -C C ss CBMC explain PBS

49
Discussion Usefulness of Fault Localization Techniques – Effectiveness: Precision: Low false negative Informative-ness: Enough clue to make a fix or refute – Efficiency: Performance: It should run within the budget constraints. Scalability: Ability to run on real size programs. Information Usage: Making the most of the information available.

50
Discussion Fault Localization Program Test Cases Specification Development History Developers Comments Input Suspicious components

51
Discussion Fault Localization Output Suspicious components Why? Program Specification No answer! …

52
Thank you!

53
What model checking gives us? We can query program (sub)paths with different characteristics. E.g. – All failing paths – All passing paths

54
The Distance Metric d An SSA-form oddity: – Distance metric can compare values from code that doesn’t run in either execution being compared – This can be the determining factor in which of two traces is most similar to a counterexample – Counterintuitive but not necessarily incorrect: simply extends comparison to all hypothetical control flow paths

55
Model Checking Model checking problem: – Given a transition system M and a property , verify if M satisfies . M can represent a program. can denote a desired property for the program, e.g.: – Deadlock does not happen, a particular function is called at most once. Model checking procedure must either verify the program or return a counter-example (failing trace).

56
What model checking gives us? We can query program (sub)paths with different characteristics. E.g. – All failing paths – All passing paths

57

58
Explanation with Distance Metrics Model checker BMC/constraint generator P+spec C Optimization tool S -C C ss CBMC explain PBS

59
Typical State of program

60

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google