1. © 2015 Carnegie Mellon University Interpolating Property Directed Reachability Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Arie Gurfinkel and Yakir Vizel July 18, 2015

2. 2 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM-0002611

3. 3 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University http://arieg.bitbucket.org/avy/

4. 4 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Verification by Successive Under-Approximation Inductive? No BMC bound 1 bound 2 bound 3

5. 5 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University INIT Reachability Analysis 5 Bad Is Bad reachable? R1R1 R2R2 …R n

6. 6 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Outline Interpolating Model Checking IC3 / Property Directed Reachabilty Avy: Interpolating Property Directed Reachability DRUP Interpolants Fast Interpolating BMC Future Directions

7. 7 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Interpolating Model Checking Introduced by McMillan in 2003 Kenneth L. McMillan: Interpolation and SAT-Based Model Checking. CAV2003: 1-13 based on pairwise Craig interpolation Extended to sequences and DAGs Yakir Vizel, Orna Grumberg: Interpolation-sequence based model checking. FMCAD 2009: 1-8 – uses interpolation sequence Kenneth L. McMillan: Lazy Abstraction with Interpolants. CAV 2006: 123-136 – IMPACT: interpolation sequence on each program path Aws Albarghouthi, Arie Gurfinkel, Marsha Chechik: From Under- Approximations to Over-Approximations and Back. TACAS 2012: 157-172 – UFO: interpolation sequence on the DAG of program paths Key Idea turn SAT/SMT proofs of bounded safety to inductive traces repeat forever until a counterexample or inductive invariant are found

8. 8 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IMC: Interpolating Model Checking N=1 BMC N SeqItp trace F = [F 0, …, F N ] Is F closed N:=N+1 CEX SAFE SAT UNSAT Yes No

9. 9 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Programs, Safety, Cexs, Invariants A transition system P = (V, Init, Tr, Bad) P is UNSAFE if and only if there exists a number N s.t. P is SAFE if and only if there exists a safe inductive invariant Inv s.t. Inductive Safe

10. 10 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Bounded Model Checking INIT R1R1 R2R2 …… INIT(V 0 ) RkRk ∧ Tr(V 0,V 1 ) ∧ … ∧ Tr(V k-1,V k ) ∧ Bad(V k )

11. 11 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Inductive Trace An inductive trace of a transition system P = (V, Init, Tr, Bad) is a sequence of formulas [F 0, …, F N ] such that Init  F 0 8 0 · i < N, F i (v) Æ Tr (v, u)  F i+1 (u) A trace is safe iff 8 0 · i · N, F i  :Bad A trace is monotone iff 8 0 · i < N, F i  F i+1 A trace is closed iff 9 1 · i · N, F i  (F 0 Ç … Ç F i-1 ) A transition system P is SAFE iff it admits a safe closed trace

12. 12 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University INIT Inductive Trace in Pictures 12 Bad F1F1 F2F2 …F N

13. 13 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Craig Interpolation Theorem Theorem (Craig 1957) Let A and B be two First Order (FO) formulae such that A ) :B, then there exists a FO formula I, denoted ITP(A, B), such that A ) I I ) :B atoms(I) 2 atoms(A) Å atoms(B) A Craig interpolant ITP(A, B) can be effectively constructed from a resolution proof of unsatisfiability of A Æ B In Model Cheching, Craig Interpolation Theorem is used to safely over- approximate the set of (finitely) reachable states

14. 14 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University A Craig Interpolant 14 B I

15. 15 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Craig Interpolant as a Circuit Let F = A(x, z) Æ B(z, y) be UNSAT, where x and y are distinct Note that for any assignment v to z either – A(x, v) is UNSAT, or – B(v, y) is UNSAT An interpolant is a circuit I(z) such that for every assignment v to z I(v) = A only if A(x, v) is UNSAT I(v) = B only if B(v, y) is UNSAT A proof system S has a feasible interpolation if for every refutation ¼ of F in S, F has an interpolant polynomial in the size of ¼ propositional resolution has feasible interpolation extended resolution does not have feasible interpolation

16. 16 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University )))))) Interpolation Sequence Given a sequence of formulas A = {A i } i=0 n, an interpolation sequence ItpSeq(A) = {I 1, …, I n-1 } is a sequence of formulas such that I k is an ITP (A 0 Æ … Æ A k-1, A k Æ … Æ A n ), and 8 k<n. I k Æ A k +1 ) I k+1 A 0 A 1 A 2 A 3 A 4 A 5 A 6 I 0 I 1 I 2 I 3 I 4 I 5 Can compute by pairwise interpolation applied to different cuts of a fixed resolution proof (very robust property of interpolation)

17. 17 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University From Interpolants to Traces A Sequence Interpolant of a BMC instance is an inductive trace ( Init(v 0 ) ) 0 Æ ( Tr (v 0,v 1 ) ) 1 Æ … Æ ( Tr (v N-1, v N ) ) N Æ Bad(v N ) F 0 (v 0 ) F 1 (v 1 ) F N (v N ) A trace computed by a sequence interpolant is safe NOT necessarily monotone NOT necessarily closed BMC N trace

18. 18 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University INIT Inductive Trace in Pictures 18 Bad F1F1 F2F2 …F N

19. 19 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University ImcMkSafe IMC: Interpolating Model Checking N=1 BMC N SeqItp trace F = [F 0, …, F N ] Is F closed N:=N+1 CEX SAFE SAT UNSAT Yes No

20. 20 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IMC: Strength and Weaknesses Strength elegant global bounded safety proof many different interpolation algorithms available easy to extend to SMT theories Weaknesses the naïve version does not converge easily – interpolants are weaker towards the end of the sequence not incremental – no information is reused between BMC queries size of interpolants hard to guide

21. 21 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IC3: Property Directed Reachability IC3: A SAT-based Hardware Model Checker Incremental Construction of Inductive Clauses for Indubitable Correctness A. Bradley: SAT-Based Model Checking without Unrolling. VMCAI 2011 PDR: Explained and extended the implementation Property Directed Reachability N. Eén, A. Mishchenko, R. K. Brayton: Efficient implementation of property directed reachability. FMCAD 2011 Very active area of research Key Idea: carefully manage SAT solving while building an inductive proof one inductive lemma at a time

22. 22 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IC3/PDR F = [Init] MkSafe Push 9 i, F i = F i+1 G = [G 0, …, G N ] F = [F 0, …, F N ] PDR trace CEX SAFE Yes No

23. 23 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University PDR Trace Recall that an inductive trace of a transition system P = (V, Init, Tr, Bad) is a sequence of formulas [F 0, …, F N ] such that Init  F 0 8 0 · i < N, F i (v) Æ Tr (v, u)  F i+1 (u) A trace is clausal if every F i is in CNF A delta-compressed trace (or ±-trace) is a sequence of clauses s.t. each clause c belongs to a unique frame F i 8 0 · i · n, 8 j < i, 8 c 2 F i. c  F j A PDR trace is a monotone, clausal, safe (up to N-1) PDR trace is often represented compactly by a ±-trace

24. 24 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IC3/PDR in Pictures PdrMkSafe

25. 25 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University IC3/PDR in Pictures Cex Queue Trace Frame F 0 Frame F 1 lemma cex PdrMkSafe

26. 26 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Inductive IC3/PDR in Pictures PdrPush

27. 27 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Inductive IC3/PDR in Pictures PdrPush PDR Invariants F i  : Bad Init  F i F i  F i+1 F i Æ Tr  F i+1 PDR Invariants F i  : Bad Init  F i F i  F i+1 F i Æ Tr  F i+1

28. 28 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University PDR Strength and Weaknesses Strengths elegant incremental many opportunities for guidance – fine-grained proof management – fine-grained generalization of lemmas Weaknesses local backward search for a counterexample CNF explosion

29. 29 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University AVY: Interpolating PDR This talk Yakir Vizel, Arie Gurfinkel: Interpolating Property Directed Reachability. CAV 2014: 260-276 Key Idea combine global BMC reasoning of IMC with local strengthening of IC3/PDR use interpolation for PDR use PDR for interpolation

30. 30 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Avy: Interpolating PDR Bounded verification with BMC Global trace using sequence interpolation Locally convert (and strengthen) to PDR trace Re-use old trace G in new BMC step Compute strengthening of old trace G by interpolation Bounded verification with BMC Global trace using sequence interpolation Locally convert (and strengthen) to PDR trace Re-use old trace G in new BMC step Compute strengthening of old trace G by interpolation N=1 BMC N SeqItp trace F = [F 0, …, F N ] 9 i, G i = G i+1 N:=N+1 CEX SAFE SAT UNSAT Yes No MkPdrTrace PDR trace G = [G 0, …, G N ] G = [G 0, …, G N ]

31. 31 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Extending a Trace Incrementally Input: A transition system P=(Init,Tr,Bad); a clausal trace F= [F 0, …, F N ] Problem: Find (if possible) a stronger safe trace G=[G 0, …, G N ] Init(v 0 ) Æ Tr (v 0,v 1 ) Æ … Æ Tr (v N-1, v N ) Æ Bad(v N ) F0F0 F1F1 FNFN F N-1 G0G0 G1G1 GNGN G N-1

32. 32 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Extending a Trace Incrementally Input: A transition system P=(Init,Tr,Bad); a clausal trace F= [F 0, …, F N ] Problem: Find (if possible) a stronger safe trace G=[G 0, …, G N ] 1.Let  = (F 0 Æ Tr 0 ) 0 Æ (F 1 Æ Tr 1 ) 1 … Æ (F N Æ Bad N ) N 2.if  is SAT then return [ ] 3.I 1, …, I n = SequenceItp (  ) 4.G 0 = Init, 8 1 · i · N. G i = F i Æ I i 5.return [G 0, …, G N ]

33. 33 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Monotone Traces by Interpolation Input: A transition system P=(Init,Tr,Bad); a safe trace F= [F 0, …, F N ] Problem: Find (if possible) a monotone safe trace G=[G 0, …, G N ] Solution: Take a sequence G 0 = Init G 1 = Itp (Init’ Ç (Init Æ Tr), : (Init’ Ç F’ 1 ) ) … G i = Itp (G’ i-1 Ç (G i-1 Æ Tr), : (G’ i-1 Ç F’ i ) ) Claim: G = [G 0, …, G N ] is a monotone and safe trace G i  G i+1 G i  : Bad G i Æ Tr  G’ i+1 G i  Ç {F j | 0 · j · i }

34. 34 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University The Tricky Part of the Proof Given a sequence G 0 = Init G 1 = Itp (G’ 0 Ç (G 0 Æ Tr), : (G’ 0 Ç F’ 1 ) G 2 = Itp (G’ 1 Ç (G 1 Æ Tr), : (G’ 1 Ç F’ 2 ) … Need to show that G 1 Æ Tr  (G’ 1 Ç F’ 2 ) by property of interpolation G 1  (G 0 Ç F 1 ) because F is a trace, F 1 Æ Tr  F’ 2 by property of interpolation G 0 Æ Tr  G’ 1 BUT the trace G=[G 0, …, G N ] is not monotone and likely to be large

35. 35 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Using PDR for Interpolation Given mutually unsatisfiable pair of formulas A and B Construct a SAFE transition system P = (A, ID, B) with initial state A transition relation ID over common variables of A and B – ID = Æ { x=x’ | x 2 Vars (A) Å Vars (B) } bad states B Run PDR/IC3 on P Claim: The frame F 1 is a CNF interpolant between A and B A Æ ID  F’ 1 == A  F 1 F 1  :B

36. 36 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Extending Monotone Clausal Traces by PDR Given a PDR trace F = [Init, F 1 ] of transition system P = (Init, Tr, Bad) G 2 -- an over over-approximation of the forward image of F 1 i.e., F 1 Æ Tr  G’ 2 Construct SAFE transition system T = (Init, Tr, Bad) where Bad = : (G 2 Ç F 1 ) Run PDR on T starting with a trace [Init, F 1, True] Claim: The sequence [Init, F 1, F 2 ] is a SAFE PDR trace

37. 37 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Extending a Trace by PDR Observations: [Init, F 1, F 2 ] is a PDR trace F 2 is stronger than G 2 Ç F 1 F 1 after is stronger than F 1 before!!! Frame F 0 Frame F 1 PdrMkSafe Frame F 2 : (G 2 Ç F 1 )

38. 38 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Avy global trace reuse prev. frame strengthen curr. trace strengthen future trace syntactic termination

39. 39 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University What is a “good” bounded proof? Proof size is not a good indicator the smallest resolution proof is usually not good – depends too much on the initial state – depends too much on the bound A “good” proof is abstract works for many ‘similar’ transition systems A proof is “good” if it extends a previously good proof re-uses existing facts

40. 40 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Searching for a “good” proof min-suffix strategy incrementally “cut” the wires to find the proof with the shortest suffix min-core strategy let SAT solver find the smallest number of wires needed for UNSAT Need better support for expressing priorities over cores!!! F0F0 F0F0 F1F1 F1F1 F2F2 F2F2 assumption for wires assumption for a frame

41. 41 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Experiments Started with an implementation based on ABC slightly modified PDR engine with external API added Sequence Interpolation SAT solving with MiniSAT and Glucose search for a good proof with one solver re-solve to compute interpolants Performs differently from PDR virtual best is much better than either one in isolation

42. 42 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Results from HWMCC’14

43. 43 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University DRUPing for Interpolants A CDCL proof is build out of trivial resolutions terminated by a learned clause A sub-proof for each learned clause can be re-constructed in polynomial time negation of clause + BCP leads to a conflict A clausal proof is a sequence of learned clauses in the order they are learned Interpolate while replaying the proof learned clause trivial resolution Arie Gurfinkel, Yakir Vizel: DRUPing for interpolats. FMCAD 2014: 99-106

44. 44 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University MiniDRUP SAT with DRUP proofs Interpolation-oriented BCP in Trim Learn near CNF interpolants in Replay SAT Trim Replay CNF Clausal Proof core proof Interpolant BCP BCP +Learning

45. 45 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Fast BMC and Interpolation State-of-the-art in Bounded Model Checking (Fast BMC) each successive bound is exponentially harder to solve many advancement in SAT since first BMC many BMC-specific advancements – circuit-aware simplifications (sweeping, constant propagation, etc.) – use of incremental SAT for increasing verification depth – lazy addition of constraints (incremental cone-of-influence) BMC used in IMC/Avy is different than BMC used for BMC interpolation algorithms assume naïve BMC circuit-aware simplifications change the structure of the formula – no correspondence between constraints and circuit steps! incremental SAT makes interpolation more difficult – many SAT queries, but one proof – what to log? Yakir Vizel, Arie Gurfinkel, Shard Malik: Fast Interpolating BMC. CAV 2015.

46. 46 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Future Directions Extending to theories easy for theories with existing interpolation procedures BUT, still need PDR-like interpolation procedure Extending to programs DAG extension for handling CFG is straight forward handling procedures (non-linear Horn clauses) is tricky – no efficient BMC. inlining == exponential explosion Many implementation decisions remain unexplored other metrics for ‘goodness’ of bounded proofs (i.e., sequence interpolants) – and corresponding proof optimization procedures switching between PDR and IMC tactics searching for a CNF interpolant vs adapting a given one

47. 47 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University http://arieg.bitbucket.org/avy/

48. 48 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Inductive Generalization A clause  is inductive relative to F iff Init   (Initialization) and  Æ F Æ Tr   ’ (Inductiveness) Implemented by first letting  = :m and generalizing  by iteratively dropping literals while checking the inductiveness condition Theorem: Let F 0, F 1, …, F N be a valid IC3 trace. If  is inductive relative to F i, 0 · i < N, then, for all j · i,  is inductive relative to F j. Follows from the monotonicity of the trace – if j < i then F j  F i – if F j  F i  then (  Æ F i Æ Tr   ’)  (  Æ F j Æ Tr   ’)

49. 49 Avy Arie Gurfinkel, July 2015 © 2015 Carnegie Mellon University Contact Information Arie Gurfinkel, Ph. D. Sr. Researcher CSC/SSD Telephone: +1 412-268-5800 Email: info@sei.cmu.edu U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Web www.sei.cmu.edu www.sei.cmu.edu/contact.cfm Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257