1. © Anvesh Komuravelli Spacer Model Checking with Proofs and Counterexamples Anvesh Komuravelli Carnegie Mellon University Joint work with Arie Gurfinkel, Sagar Chaki, and Edmund Clarke

2. © Anvesh Komuravelli Spacer Safety of Programs Program P + Safety Assertions Program P + Safety Assertions Automatic verification for assertion failures Automatic verification for assertion failures Safe + Proof Unsafe + Counterexample 2 0. x := 0; 1.while (x < n) { 1. x := x + 1; } 3. x := -x; 4. assert (x ≤ 0) // x ≥ 0 Loop Invariant

3. © Anvesh Komuravelli Spacer Safety of Programs Program P + Safety Assertions Program P + Safety Assertions Automatic verification for assertion failures Automatic verification for assertion failures Safe + Proof Unsafe + Counterexample 3 While-programs (unbounded variables, no procedure calls, no memory operations) While-programs (unbounded variables, no procedure calls, no memory operations) 0. x := 0; 1.while (x < n) { 2. x := x + 1; } 3. x := -x; 4. assert (x ≤ -1) Counterexample Trace: pc=0. x=0, n=0 pc=1. else branch pc=3. x=0 pc=4. error Counterexample Trace: pc=0. x=0, n=0 pc=1. else branch pc=3. x=0 pc=4. error

4. © Anvesh Komuravelli Spacer Algorithms for Safety 4 1.Safety is undecidable! Reduction from Halting Problem to safety of a 2-counter machine 2.Existing algorithms use heuristics for verifying many programs in practice 3.Two broad classes of model checking algorithms: A.Generalize feasible and safe behaviors (Proof-Based) B.Eliminate infeasible and unsafe behaviors (Counterexample-based) 1.Safety is undecidable! Reduction from Halting Problem to safety of a 2-counter machine 2.Existing algorithms use heuristics for verifying many programs in practice 3.Two broad classes of model checking algorithms: A.Generalize feasible and safe behaviors (Proof-Based) B.Eliminate infeasible and unsafe behaviors (Counterexample-based) This talk: Improve (A) based on ideas from (B)

5. © Anvesh Komuravelli Spacer 5 Background on Proof-Based algorithms

6. © Anvesh Komuravelli Spacer should never hold Transition System Programs as Transition Systems 6 Variables Init conditionTransition relation Error condition 0. x := 0; 1. while (x < n) { 2. x := x + 1; } 3. x := -x; 4. assert (x ≤ 0) encodes how data and control change after every instruction encodes how data and control change after every instruction “ “

7. © Anvesh Komuravelli Spacer SAT-Based Model Checking 7 Counterexample of length 0? Counterexample of length 1? SAT? … Bounded Model Checking, Clarke et al., TACAS 1999 Transition System 1.Boolean SAT is NP-complete, but we have efficient solvers today 2.SAT modulo theories (SMT) for handling arithmetic, etc. 3.Eg: is unsatisfiable for integers x, y 1.Boolean SAT is NP-complete, but we have efficient solvers today 2.SAT modulo theories (SMT) for handling arithmetic, etc. 3.Eg: is unsatisfiable for integers x, y

8. © Anvesh Komuravelli Spacer SAT-Based Model Checking 8 … Bounded Model Checking, Clarke et al., TACAS 1999 Transition System No upper bound on the length of a counterexample! Even for finite-state systems, the upper bound can be huge! No upper bound on the length of a counterexample! Even for finite-state systems, the upper bound can be huge! When do we stop? Are initial states safe? Are 1 step-reachable states safe? Keep track of the reachable states! Keep track of the reachable states! Counterexample of length 0? Counterexample of length 1? SAT?

9. © Anvesh Komuravelli Spacer … Keep Track of the Reachable States 9 err(x) reach(P) Initial States States reachable in ≤1 steps States reachable in ≤2 steps Usually Hopeless!

10. © Anvesh Komuravelli Spacer Reachable states can be diverging! 10 0. x := 0; 1. while (x < n) { 2. x := x + 1; } 3. x := -x; 4. assert (x ≤ 0) … reachable states at ( pc =1) n is a symbolic input (diverging) converged!

11. © Anvesh Komuravelli Spacer err(x) Generalize (Heuristics using Craig Interpolation [1,2] ) Generalize the reachable states! 11 [1] McMillan, Interpolation and SAT-Based Model Checking, CAV 2003 [2] McMillan, Lazy Abstraction with Interpolants, CAV 2006 …

12. © Anvesh Komuravelli Spacer err(x) Generalize the reachable states! 12 … reach(P)

13. © Anvesh Komuravelli Spacer Proofs and Invariants 13 0. x := 0; 1. while (x < n) { 2. x := x + 1; } 3. x := -x; 4. assert (x ≤ 0) … reachable states at ( pc =1) (diverging) x ≥ 0 is a loop invariant! x ≥ 0 is a loop invariant! The actual set of reachable states is stronger: 0 ≤ x ≤ n The actual set of reachable states is stronger: 0 ≤ x ≤ n Proof of Safety Proof of “Bounded” Safety Proof of “Bounded” Safety

14. © Anvesh Komuravelli Spacer Many heuristics for generalizations! 14 err(x) No unique generalization! Today’s best algorithms for hardware verification are SAT-based Several competitive algorithms exist for software No unique generalization! Today’s best algorithms for hardware verification are SAT-based Several competitive algorithms exist for software One possible generalization Another possible generalization

15. © Anvesh Komuravelli Spacer 15 The Problem Generalizations are not always sufficient

16. © Anvesh Komuravelli Spacer Generalizations can suffer from local view x = y = z = w = 0; while (*) { // loop invariant: // (x ≥ 4 => y ≥ 100) && (z ≤ 10w) if (*) {x++; y += 100;} else if (*) if (x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) { y = −y; } t = 1; w += t; z += 10t; } assert (!(x ≥ 4 && y ≤ 2)) State-of-the-art Tool Z3 cannot verify in an hour Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08. 16 Proofs of Bounded Safety never connect z and w Proofs of Bounded Safety never connect z and w

17. © Anvesh Komuravelli Spacer Abstractions for better generalizations! x = y = z = w = 0; while (*) { // loop invariant: // (x ≥ 4 => y ≥ 100) && (z ≤ 10w) if (*) {x++; y += 100;} else if (*) if (x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) { y = −y; } t = 1; w += t; z += 10t; } assert (!(x ≥ 4 && y ≤ 2)) State-of-the-art Tool Z3 cannot verify in an hour Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08. 17 Verifies the abstraction in < 1 sec. t = *; Abstractions only add behaviors

18. © Anvesh Komuravelli Spacer How to obtain helpful abstractions automatically? 18 1.An abstraction of the program can dramatically improve generalizations! 2.How to obtain helpful abstractions? 3.How to efficiently and automatically maintain abstractions? 1.An abstraction of the program can dramatically improve generalizations! 2.How to obtain helpful abstractions? 3.How to efficiently and automatically maintain abstractions?

19. © Anvesh Komuravelli Spacer Spurious counterexample CounterExample- Guided Abstraction Refinement (CEGAR) [1] CounterExample- Guided Abstraction Refinement (CEGAR) [1] err(x) [1] Clarke et al., Counterexample-Guided Abstraction Refinement, CAV 2000. reach(P) Abstractions are great, but not always! 19 Reachable states of an abstraction The second class of algorithms

20. © Anvesh Komuravelli Spacer 20 Our algorithm Spacer

21. © Anvesh Komuravelli Spacer Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample 21

22. © Anvesh Komuravelli Spacer Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) 22 Proofs from Abstractions

23. © Anvesh Komuravelli Spacer Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample 23 Refinement using Spurious Counterexamples

24. © Anvesh Komuravelli Spacer Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) 24 Proof-Based Abstraction

25. © Anvesh Komuravelli Spacer Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample 25

26. © Anvesh Komuravelli Spacer Initial States of P Initial States of P Generalization/Proof err(x) reach(P) reach(A 1 ) Spacer at a high level 26 Initial States of A 1 Initial States of A 1 Abstraction Proof-Based Abstraction

27. © Anvesh Komuravelli Spacer err(x) reach(P) reach(A 1 ) Reachable states of A 1 in ≤1 steps Spurious counterexample Spacer at a high level 27

28. © Anvesh Komuravelli Spacer err(x) reach(P) reach(A 2 ) Spacer at a high level Reachable states of A 2 in ≤1 steps Refine A 1 to A 2 eliminating the spurious counterexample 28 Generalization/ Proof Reachable states of P in ≤1 steps

29. © Anvesh Komuravelli Spacer err(x) reach(P) reach(A 3 ) Spacer at a high level Reachable states of A 3 in ≤1 steps 29 Reachable states of P in ≤1 steps Fresh abstraction, to avoid bias Fresh abstraction, to avoid bias Proof-Based Abstraction

30. © Anvesh Komuravelli Spacer Key Ideas of Spacer 30 1.Abstractions help obtain (hopefully) more general proofs 2.First integration of Proof-Based Abstraction with SAT/SMT-Based Model Checking 3.Orthogonal to heuristics for Interpolation/Generalization 1.Abstractions help obtain (hopefully) more general proofs 2.First integration of Proof-Based Abstraction with SAT/SMT-Based Model Checking 3.Orthogonal to heuristics for Interpolation/Generalization Implementation and Experimental Evidence on C Programs

31. © Anvesh Komuravelli Spacer Abstractions add a new dimension 31 … … … … SAT-Based Model Checking AbstractAbstract

32. © Anvesh Komuravelli Spacer SAT-Based Model Checking with Abstractions 32 … … … … Under-approximations AbstractAbstract need not be monotonic

33. © Anvesh Komuravelli Spacer SAT-Based Model Checking with Abstractions 33 … … … … Under-approximations AbstractAbstract non-trivial abstraction

34. © Anvesh Komuravelli Spacer Spacer (Software Proof-based Abstraction with CounterExample-based Refinement) Program Fix a Bound Check Safety Feasible? Invariants? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample 34

35. © Anvesh Komuravelli Spacer 35 Spacer on Example

36. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; while (*) { // (y > 10w) => (z < 100x), z ≤ 100x, // x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1 assume (c < 2); if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w++; z += 10; c += 1; } assert (!(x ≥ 4 && y ≤ 2)); Add Counters Bound Solve  Loop Invariants 36

37. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; while (*) { // (y > 10w) => (z < 100x), z ≤ 100x, // x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1 assume (c < 2); if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w++; z += 10; c += 1; } assert (!(x ≥ 4 && y ≤ 2)); BoundSolve  Unbounded?  Preserved! Specific to under-approx. Depend on counter Extract Unbounded Invariants 37 Treat as conjectured unbounded invariants. (as in Houdini [1] ). Treat as conjectured unbounded invariants. (as in Houdini [1] ). [1] Houdini, an annotation assistant for ESC/Java, C. Flanagan and K.R.M. Leino, 2001

38. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; while (*) { // (y > 10w) => (z < 100x), z ≤ 100x, // x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1 if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w++; z += 10; c += 1; } assert (!(x ≥ 4 && y ≤ 2)); Bound Solve  Unbounded?  NO Invariants are too weak! Invariants are too weak! 38 Abstract  ✖

39. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; assume (y > 10w => z < 100x, z ≤ 100x); while (*) { // (y > 10w) => (z < 100x), z ≤ 100x, // x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1 assume (c < 2); if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w++; z += 10; c += 1; assume (y > 10w => z < 100x, z ≤ 100x); } assert (!(x ≥ 4 && y ≤ 2)); Abstract Redundant 39 Bound Solve  Unbounded?  NO  Strengthen with Invariants

40. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; assume (y > 10w => z < 100x, z ≤ 100x); while (*) { // (y > 10w) => (z < 100x), z ≤ 100x, // x ≤ 2, c ≤ 0 => x ≤ 0, c ≤ 1 => x ≤ 1 assume (c < 2); if (*) {x++; y = *;} else if (* && x ≥ 4) {x++; y = *;} else if (y > 10w && z ≥ 100x) {y = *;} else assume (false); w = *; z = *; c += 1; assume (y > 10w => z < 100x, z ≤ 100x); } assert (!(x ≥ 4 && y ≤ 2)); 40 BoundAbstract Solve  Unbounded?  NO  Proof-Based Abstraction

41. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; assume (y > 10w => z < 100x, z ≤ 100x); while (*) { assume (c < 4); if (*) {x++; y = *;} else if (* && x ≥ 4) {x++; y = *;} else if (y > 10w && z ≥ 100x) {y = *;} else assume (false); w = *; z = *; c += 1; assume (y > 10w => z < 100x, z ≤ 100x); } assert (!(x ≥ 4 && y ≤ 2)); Bound Solve  Counterexample! Increment x to 4 Choose y arbitrarily Feasible?  Concrete control path is infeasible NORefine  Concretize 41

42. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; assume (y > 10w => z < 100x, z ≤ 100x); while (*) { assume (c < 4); if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w = *; z = *; c += 1; assume (y > 10w => z < 100x, z ≤ 100x); } assert (!(x ≥ 4 && y ≤ 2)); BoundSolve  Feasible?  NORefine  42 CEGAR

43. © Anvesh Komuravelli Spacer Spacer on Example x = y = z = w = 0; c = 0; assume (y > 10w => z < 100x, z ≤ 100x); while (*) { // (y > 10w) => (z < 100x), z ≤ 100x // y > 0, (x > 0) => (y ≥ 100) assume (c < 4); if (*) {x++; y += 100;} else if (* && x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) {y = −y;} else assume (false); w = *; z = *; c += 1; assume (y > 10w => z < 100x, z ≤ 100x); } assert (!(x ≥ 4 && y ≤ 2)); Bound Solve  Unbounded?  YES Invariants 43

44. © Anvesh Komuravelli Spacer 44 Implementation Details

45. © Anvesh Komuravelli Spacer Three Key Components 45 1.Extracting Unbounded Invariants 2.Proof-Based Abstraction 3.Counterexample Analysis and Refinement 1.Extracting Unbounded Invariants 2.Proof-Based Abstraction 3.Counterexample Analysis and Refinement How can we efficiently use today’s SAT/SMT solvers? Focus

46. © Anvesh Komuravelli Spacer An invariant for the transition system is a formula that holds for the initial states and after every transition An invariant for the transition system is a formula that holds for the initial states and after every transition Extracting Unbounded Invariants 46 φ is an invariant iff Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t. Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t. (initial) (transition)

47. © Anvesh Komuravelli Spacer SAT? Re-start from scratch! Extracting Unbounded Invariants 47 unsatsat, making true … until fixed point Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t. Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t.

48. © Anvesh Komuravelli Spacer Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t. Given: A set L of conjectures for invariants, each including “initial states”. Goal: A maximal I L s.t. Introduce assumption variables Extracting Unbounded Invariants 48 unsatsat, making true …

49. © Anvesh Komuravelli Spacer Proof of Bounded Safety Proof of Bounded Safety Not all of is necessary! Proof-Based Abstraction 49 A proof of “bounded” safety for is a formula that holds of the initial states and after every transition up to the given bound, and excludes error states. A proof of “bounded” safety for is a formula that holds of the initial states and after every transition up to the given bound, and excludes error states. (initial) (transition) (safety)

50. © Anvesh Komuravelli Spacer unsat Proof-Based Abstraction 50 unsat UNSAT core Iteratively minimize

51. © Anvesh Komuravelli Spacer What have we seen so far? 51 1.Generalizing reachable states can be hard! 2.Abstractions can really help! 3.Algorithm Spacer that combines abstraction refinement with SAT-based model checking 4.How Spacer can be efficiently automated 1.Generalizing reachable states can be hard! 2.Abstractions can really help! 3.Algorithm Spacer that combines abstraction refinement with SAT-based model checking 4.How Spacer can be efficiently automated

52. © Anvesh Komuravelli Spacer 52 Tool and Experimental Evaluation

53. © Anvesh Komuravelli Spacer Spacer Tool 53 C Program (Horn-SMT) Logical Encoding Spacer Backend (using Z3’s framework) Existing Front-end based on LLVM Proof-Based Abstraction, CEGAR, etc. Theories handled: Linear Arithmetic (Rationals and Integers), Bitvectors Theories handled: Linear Arithmetic (Rationals and Integers), Bitvectors

54. © Anvesh Komuravelli Spacer Spacer Tool Program Under- Approximate Check Safety Feasible? Abstract Refine Proof-Based Abstraction CEGAR No Yes Safety ProofCounterexample 54 SMT-Based Model Checker in Z3 SMT-Based Model Checker in Z3

55. © Anvesh Komuravelli Spacer The hard example mentioned in the beginning x = y = z = w = 0; while (*) { if (*) {x++; y += 100;} else if (*) if (x ≥ 4) {x++; y++;} else if (y > 10w && z ≥ 100x) { y = −y; } t = 1; w += t; z += 10t; } assert (!(x ≥ 4 && y ≤ 2)) State-of-the-art Tool Z3 cannot verify in an hour Source: Automatically Refining Abstract Interpretations, Gulavani, Chakraborty, Nori and Rajamani, TACAS ‘08. 55 Spacer automatically verifies in under a minute! Spacer automatically verifies in under a minute!

56. © Anvesh Komuravelli Spacer Results on SV-COMP’13 Benchmarks 56 Advantage! Time-out Mem-out

57. © Anvesh Komuravelli Spacer 57 Summary

58. © Anvesh Komuravelli Spacer Conclusion Focused Proofs: Abstractions guide Interpolation towards certain generalizations Combine Proof-Based Abstraction and Counterexample-Guided Refinement General framework independent of the underlying model checker Works in practice! Focused Proofs: Abstractions guide Interpolation towards certain generalizations Combine Proof-Based Abstraction and Counterexample-Guided Refinement General framework independent of the underlying model checker Works in practice! Future Directions Verification in presence of assumptions Different kinds of bounding/abstraction Synthesizing ghost code to help verification Verification in presence of assumptions Different kinds of bounding/abstraction Synthesizing ghost code to help verification

59. © Anvesh Komuravelli Spacer Questions? 59 For more details, read our CAV’13 paper!

60. © Anvesh Komuravelli Spacer Concrete: Abstract: Counterexample Analysis and Refinement 60 An “abstract counterexample” is a finite length path consistent with error states SAT SAT ? Feasibility Check