1. SAT-based Model Checking Yakir Vizel Computer Science Department, Technion, Israel Based on slides from K.L. McMillan, A.R. Bradley and Yakir Vizel

2. Outline Background –Symbolic Model Checking –DPLL-style SAT solvers –Bounded Model Checking SAT-based model checking methods –K-induction –Interpolation –Interpolation Sequence –IC3/PDR –There are more…

3. Model checking Problem definition: –Does every run of a (finite-state) transition system satisfy a given temporal property? Result: –Yes –No + counterexample Examples: –Is every request to this bus arbiter eventually acknowledged? –Does this program every dereference a null pointer?

4. Transition systems Tuple (S,I,T), where: –S is the (finite) set of states –I  S is the set of initial states –T  S  S is the set of transitions A run of (S,I,T) is  S , where: –  0  I –for all i  0, (  i,  i+1 )  T That is, a run is an infinite path in the state graph strating with an initial state

5. Reachability Problem def: –Does a transition system have a finite run ending in a state contained in the failure set F? –More precisely, does there exist  0...  k  S k s.t.:  0  I and  k  F for all 0  i < k, (  i,  i+1 )  T Using automata-theoretic methods, model checking safety properties reduces to reachability analysis. –Given a t.s. M and a property P, we can construct M P and F P such that M satisfies P exactly when F P is not reachable in M P.

6. State explosion problem Reachability analysis can be done by BFS or DFS on the state graph. However, |S| is exponential in system size –for example 2 n, where n is number of registers Impractical to construct the state graph explicitly. Our topic is essentially how to use a SAT solver to tackle this problem.

7. Symbolic transition systems Tuple (V,I,T), where: –V is a signature (set of variables), –I is a formula over V (the initial condition) –T is a formula over V  V' (the transition condition) States:  = V  {0,1} (a valuation to V) A run of (V,I,T) is  , where: –I[  0 ] –for all i  0, T[  i,  i+1 ] Note: T[  i,  i+1 ] means T[  i  ' i+1 ]

8. Example T is a conjunction of constraits, one per component. a b cp g g = a  b p = g  c c' = p T =  { g = a  b, p = g  c, c' = p }

9. Symbolic Reachability IF R1R1 R2R2... R = I  Img(I,T) = R 1  Img(R 1,T) Idea: represent reachable states by a formula Essentially a BFS with symbolic representation.

10. Symbolic reachability, cont. Reachability fixed point: R 0 = I R i+1 = R i  Img(R i,T) R =  R i F is reachable iff R  F  false Image operator: Img(Q,T) =  V. (Q(V)  T(V,V ’ )) We need a way to eliminate the quantifier, to get us back to an ordinary Boolean formula.

11. DPLL-style SAT solvers Objective: –Check satisfiability of a CNF formula literal: v or  v clause: disjunction of literals CNF: conjunction of clauses Approach: –Branch: make arbitrary decisions –Propagate implication graph –Use conflicts to guide inference steps SATO,GRASP,CHAFF,BERKMIN SAT solvers can also generate refutation proofs!

12. The Implication Graph (BCP) (  a  b)  (  b  c  d) a cc Decisions b Assignment: a  b   c  d d

13. Propositional Resolution a  b   c  a   c  d b   c  d When a conflict occurs, the implication graph is used to guide the resolution of clauses, so that the same conflict will not occur again.

14. Conflict Clauses (  a  b)  (  b  c  d)  (  b   d) a cc Decisions b Assignment: a  b   c  d d Conflict! (  b  c ) resolve Conflict! (  a  c) resolve Conflict!

15. Generating refutations Refutation = a proof of the null clause –Record a DAG containing all resolution steps performed during conflict clause generation. –When null clause is generated, we can extract a proof of the null clause as a resolution DAG. Original clauses Derived clauses Null clause

16. Circuit SAT a b c p g Can the circuit output be 1? input variables output variable (a   g)  (b   g)  (  a   b  g) (  g  p)  (  c  p)  (g  c   p) CNF(p) p is satisfiable when the formula CNF(p)  p is satisfiable

17. Bounded Model Checking Given –A finite transition system M –A property P Determine –Does M allow a counterexample to P of k transitions of fewer? This problem can be translated to a SAT problem BCCZ99

18. Symbolic Models (recall) Transition system described by constraints... a b cp g g = a  b p = g  c c' = p New notation: Q means "add n primes to the symbols in Q" T =  { g = a  b, p = g  c, c' = p }

19. Bounded model checking Unfold the model k times: U = T  T ...  T a b cp g a b cp g a b cp g... I F Use SAT solver to check satisfiability of I  U  F If unsatisfiable: property has no Cex of length k can produce a refutation proof P Biere,et al. TACAS99

20. Bounded Model Checking I R1R1 R2R2 = I  Img(I,T) = R 1  Img(R 1,T) F I  T  T   F ……

21. BMC applications Debugging: –Can find counterexamples using a SAT solver Proving properties: –Only possible if a bound on the length of the shortest counterexample is known. I.e., we need a diameter bound. The diameter is the maximum lenth of the shortest path between any two states. –Worst case is exponential. Obtaining better bounds is sometimes possible, but generally intractable.

22. Unbounded Model Checking We consider a variety of methods to explioit SAT and BMC for unbounded model checking: –K-step induction –Abstraction Counterexample-based Non-counterexample-based –Exact image computations SAT solver tests for fixed point SAT solver computes image –Over-approximate image computations

23. Induction The simple case: P is an inductive invariant –I => P –P  T => P’ Usually, P is not an inductive invariant BUT – a stronger inductive invariant R may exist (strengthening) –I => R –R  T => R’ –R => P

24. Induction P R I

25. K-induction Induction: SSS2000 P(s 0 )  i : P(s i )  P(s i+1 )  i : P(s i ) k-step induction: P(s 0..k-1 )  i : P(s i..i+k-1 )  P(s i+k )  i : P(s i )

26. K-induction with a SAT solver Recall: U k = T  T ...  T Two formulas to check: –Base case: I  U k-1  P...P –Induction step: U k  P...P  P If both are valid, then P always holds. If not, increase k and try again.

27. Simple path assumption Unfortunately, k-induction is not complete. –Some properties not k-inductive for any k. Simple path restriction: –There is a path to  P iff there is a simple path to  P (path with no repeated states). PP PP

28. Induction over simple paths Let simple(s 0..k ) be defined as: –  i,j in 0..k : (i  j)  s i  s j k-induction over simple paths: P(s 0..k-1 )  i : simple(s 0..k )  P(s i..i+k-1 )  P(s i+k )  i : P(s i ) Must hold for k large enough, since a simple path cannot be unboundedly long. Length of longest simple path is called recurrence diameter.

29. ...with a SAT solver For simple path restriction, let: S k =  t=0..k, u=t+1..k:   v in V : v t = v u (where V is the set of state variables). Two formulas to check: –Base case: I  U k-1  P...P –Induction step: S k  U k  P...P  P If both are valid, then P always holds. If not, increase k and try again.

30. Termination Termination condition: k is the length of the longest simple path of the form P*  P This can be exponentially longer than the diameter. –example: loadable mod 2 N counter where P is (count  2 N -1) diameter = 1 longest simple path = 2 N Nice special cases: –P is a tautology (k=0) –P is inductive invariant (k=1)

31. Image computation methods Symbolic model checking without BDD's –Use SAT solver just for fixed-point detection Abdulla, Bjesse and Een 2000 Williams, Biere, Clarke and Gupta 2000 –Adapt SAT solver to compute image directly McMillan, 2002

32. Image over-approximation BMC and Craig interpolation allow us to compute image over-approximation relative to property. –Avoid computing exact image. –Maintain SAT solver's advantage of filtering out irrelevant facts.

33. Interpolation If A  B = false, there exists an interpolant A' for (A,B) such that: A  A' A'  B = false A' refers only to common variables of A,B Example: –A = p  q, B =  q  r, A' = q New result –given a resolution refutation of A  B, A' can be derived in linear time. (Craig,57) (Pudlak,Krajicek,97)

34. Interpolation-based MC Interpolation gives us –SAT-based algorithm for over-approximate image computation, using interpolation –SAT-only symbolic model checking (McMillan,2003)

35. Reachability Is there a path from I to F satisfying transitions T? Reachability fixed point: R 0 = I R i+1 = R i  Img(R i,T) R =  R i Image operator: Img(Q,T) =  V. (Q  T) F is reachable iff R  F  false

36. Overapproximation An overapproximate image op. is Img' s.t. for all Q, Img(Q,T) implies Img'(Q,T) Overapprimate reachability: R' 0 = I R' i+1 = R' i  Img'(R' i,T) R' =  R' i Img' is adequate (w.r.t.) F, when –if Q cannot reach F, Img ’ (Q,T) cannot reach F If Img' is adequate, then –F is reachable iff R'  F  false

37. Adequate image Q F Img(Q,T) Reached from Q Can reach F Img ’ (Q,T) But how do you get an adequate Img'?

38. k-adequate image operator Img' is k-adequate (w.r.t.) F, when –if Q cannot reach F, Img ’ (Q,T) cannot reach F within k steps Note, if k > diameter, then k-adequate is equivalent to adequate.

39. Interpolation-based image Idea -- use unfolding to enforce k-adequacy A = Q  T B = T  T  T  F k F k = ¬P ∨ ¬P ∨ … ∨ ¬P Q F TTTTTTT AB t=1 t=k Let Img'(Q) 0 = A', where A' is an interpolant for (A,B)... Img' is k-adequate!

40. Given the following BMC formula. A B A’A’

41. Huh? A  A' –Img(Q,T)  Img'(Q,T) A'  B = false – Img’(Q,T) cannot reach F in k steps Hence Img' is k-adequate overapprox. Q F TTTTTTT AB t=1 t=k A' But note, Img' is partial -- not defined if A  B is sat.

42. k-adequate I F1F1 F2F2 = I  Img’(I,T) = F 1  Img’(F 1,T) P ¬P …… k k

43. A1A1 A2A2 Using Interpolants

44. A’1A’1...... Using Interpolants (2)

45. I R1R1 R1R1 I R2R2 R2R2 R3R3 R3R3 ¬P A’1A’1 A’1A’1 A’2A’2 A’2A’2 A’3A’3 A’3A’3 Analogy To Reachability Analysis

46. Reachability algorithm let k = 0 repeat if I can reach F within k steps, answer reachable R = I while Img'(R,T)  F = false R' = Img'(R,T)  R if R' = R answer unreachable R = R' end while increase k end repeat

47. Termination Since k increases at every iteration, eventually k > d, the diameter, in which case Img' is adequate, and hence we terminate. Notes: –don't need to know when k > d in order to terminate –often termination occurs with k << d

48. Interpolation-based MC Fully SAT-based. Inherits SAT solvers ability to concentrate on facts relevant to a property. Most effective when –Very large set of facts is available –Only a small subset are relevant to property For true properties, appears to converge for smaller k values.

49. Interpolation-Sequence If A 1  A 2  A k = false, there exists an interpolation-sequence A’ 0, A’ 1,…, A’ k+1 for (A 1,  A k ) such that: A’ 0 =T and A’ k+1 =F A’ j  A j+1  A’ j+1 A’ j - over common variables of A 1,  Aj and A j+1,  A k A’ j equals the interpolant of A=A 1  A j and B=A j+1  A k –Given the same resolution graph

50. 50 A1A1 A2A2 A3A3 AkAk A k+1 A’1A’1 A’2A’2 A’3A’3 A’ k-1 A’kA’k BMC formula partitioned in a different manner: Interpolation-Sequence based MC (Vizel and Grumberg,2009)

51. 51 I 1,1 I 1,2 I 2,2 I1I1 I1I1 Using Interpolation-Sequence

52. 52 INIT R1R1 R1R1 R2R2 R2R2 R3R3 R3R3 A1A1 A1A1 A A A3A3 A3A3 ¬P A 1,1 A 2,2 A 1,2 A1A1 A1A1 A2A2 A2A2 A 3,3 A 2,3 A 1,3 Analogy to Forward Reachability

53. Conclusion SAT solvers are very effective at ignoring irrelevant facts SAT solvers can produce refutations We can exploit in a number of ways: –BMC –Abstraction for UMC (either CBA or PBA) –Abstract image computations using interpolation This makes it possible to model check localizable properties large systems.

54. IC3 – The Breakthrough IC3 = Incremental Construction of Inductive Clauses for Indubitable Correctness The Goal: Find an Inductive Invariant stronger than P by learning relatively inductive facts (incrementally) –Recall: F is inductive invariant if I => F F  T => F’ –F is stronger than P, therefore F => P F  P  T => F’ => P’ (Bradley,2010)

55. What Makes IC3 Special? No unrolling/unfolding of the transition function T is required All previous approaches require unrolling –Searching for an inductive invariant –Unrolling = A form of strengthening IC3 strengthen in a different way

56. Example M 1 : –I: x=1, y=1 –T: x’= x+1, y’= y+x M 2 : –I: x=1, y=1 –T: x’= x+y, y’= y+x P: y ≥ 1

57. Induction on M 1 I => P: –x=1  y=1 => y ≥ 1 BUT: P  T ≠> P’ –y ≥ 1  x’=x+1  y’=x+y => y’ ≥ 1 Incremental Proof: –F = x ≥ 0 –x ≥ 0  x’=x+1 =>  x’ ≥ 0 –Clearly, F  P  T => P’

58. Induction on M 2 I => P: –x=1  y=1 => y ≥ 1 BUT: P  T ≠> P’ –y ≥ 1  x’=x+y  y’=x+y => y’ ≥ 1 Monolithic Proof: –Choose a new P*, stronger than P –Note that F = x ≥ 0 is also not inductive Set P* = x ≥ 0  y ≥ 1 –I => P* –P*  T => P*’

59. IC3 Basics Compute Over-approximated Reachability Sequence (OARS) s.t. –F 0 = I –F i => P –F i => F i+1 –F i  T => F’ i+1 Notations: –Cube s: conjunction of literals v 1  v 2  ¬v 3 - Represents a state –s is a cube => ¬s is a clause (DeMorgan) –P is inductive relative to F if F is inductive and F  P  T => P’

60. OARS I R1R1 R2R2 = I  Img(I,T) = R 1  Img(R 1,T) P F1F1 F2F2 ¬P

61. A Backward Search Search for a predecessor s to some error state: P  T  ¬P’ –If none exists, property holds: P  T  ¬P’ = ¬(P  T  => P’) Try to block s –P = P  ¬s

62. IC3 - Initialization Check the following two formulas: –I  ¬P’ –I  T  ¬P’ If both are unsatisfiable then: –I => P –I  T => P’ Therefore –F 0 = I, F 1 = P

63. IC3 - Initialization I F0F0 P F1F1

64. Iteration I F1F1 F2F2 P F k+1 …… FkFk F k-1

65. IC3 - Iteration Given an OARS, define F k+1 =P Apply a backward search –Find predecessor s in F k that can reach a bad state F k  T => P’ (F k  T  ¬P’) –If none exists, move to next iteration –If exists, try to find a predecessor t to s in F k-1 F k-1  T => ¬s’ (F k-1  T  s’) –If non exists, s can be removed from F k F k = F k  ¬s –Otherwise: Recur on (t,k-1) We call (t,k-1) a proof obligation If we can reach I, a CEX exists

66. That Simple? Looks simple But this “simple” does NOT work Simple = States Enumeration –Too many states… Are we enumerating states? –Yes (when it doesn’t work) –But, not really

67. Observation 1 A state s in F k can reach a bad state in one transition Important Fact: s is not in F k-1 (!!) –F k-1  T => F k –F k => P –If s was in F k-1 we would have found it in an earlier iteration Therefore: F k-2  T => ¬s’

68. Inductive Generalization A state s in F k can reach a bad state in one transition Assuming s in not reachable: –F k-1  T => ¬s’ BUT, this is stronger: F k-1  ¬s  T => ¬s’ This looks familiar! –I => ¬s Otherwise, CEX! (I ≠> ¬s  s is in I) –¬s is inductive relative to F k-1

69. Inductive Generalization So we know F k-1  ¬s  T => ¬s’ And, ¬s is a clause Generalize: Find a sub-clause c ¬s s.t. F k-1  c  T => c’ (and I => c) –Sub clause means less literals –Less literals implies less satisfying assignments (a ∨ b ∨ c) vs. (a ∨ b) –Stronger inductive fact More states are removed from F k, making it stronger (closer to R k )

70. Observation 2 A state s in F i can reach a bad state in a number of transitions s is also in F j for j > i, a longer CEX may exist –F i => F j –s may not be reachable in i steps, but it may be reachable in j steps

71. Push Forward I F1F1 F2F2 P F k+1 …… FkFk F k-1

72. Push Forward s is removed from F i –by conjoining a sub-clause c –F i = F i  c Pushing forward for j > i –F j  c  T => c’ holds –c is inductive relative to F j –F j = F j  c s cannot be blocked at level j > i –Add a proof obligation (s,j) –If s is reachable from I, CEX!

73. IC3 – Key Ingredients Backward Search –Find a state s that can reach a bad state in a number of steps –s may not be reachable (over-approximations) Block a State –Do it efficient, block more than s Generalization Push Forward –An inductive fact at frame i, may also be inductive at higher frames –If not, a longer CEX is found

74. The End