1. © Andrew IrelandDependable Systems Group Cooperative Reasoning for Automatic Software Verification Andrew Ireland Dependable Systems Group School of Mathematical & Computer Sciences Heriot-Watt University Edinburgh

2. © Andrew IrelandDependable Systems Group Outline A historical perspective New wave of software verification tools Cooperative reasoning Applications and challenges

3. © Andrew IrelandDependable Systems Group Assertion Based Verification “Assertion boxes” Goldstine & von Neuman 1947 “Checking a large routine” Turing 1949 “Inductive assertion method” Floyd 1967 “Axiomatic basis” Hoare 1969 “A program verifier” King 1969

4. © Andrew IrelandDependable Systems Group Floyd’s Summation Program

5. © Andrew IrelandDependable Systems Group Floyd’s Summation Program Pre-condition Post-condition

6. © Andrew IrelandDependable Systems Group Loop invariant Floyd’s Summation Program

7. © Andrew IrelandDependable Systems Group Floyd’s Verifying Compiler Proposal “The Verifying Compiler” CMU Annual Report p18-19, 1967

8. © Andrew IrelandDependable Systems Group First Wave of Verifiers “A Program Verifier” (King 1969) Stanford Pascal Verifier (Luckham et al 1973) Gypsy (Good et al 1974) VISTA (German & Wegbreit 1975) HDM (Levitt et al 1977) SPARK for Ada (Carre et al 1983) Penelope (Guaspari et al 1990) ESC/Java ( Detlefs et al 1998)

9. © Andrew IrelandDependable Systems Group So Why the Poor Uptake? Formalizing specifications is hard Verification requires auxiliary specifications and properties, e.g. loop invariants, inductive properties Mechanized proof is semi-automatic at best, i.e. skilled user interaction required Scalability issues associated with mainstream programming languages, e.g. reasoning about pointers

10. © Andrew IrelandDependable Systems Group Times They Are Changing Static Device Verifier (SDV): http://www.microsoft.com/whdc/devtools/tools/sdv.mspx ESC/Java2: http://secure.ucd.ie/products/opensource/ESCJava2 Spec#: http://research.microsoft.com/specsharp The SPARK Approach: http://www.praxis-his.com/sparkada

11. © Andrew IrelandDependable Systems Group What’s Changed? Generic property verification “Buffer overflows have been the most common form of security vulnerability for the past 10 years“ OGI/DARPA Programming language subsets Programming logics Automated reasoning tools Tool integrations

12. © Andrew IrelandDependable Systems Group Cooperative Reasoning +=

13. © Andrew IrelandDependable Systems Group Cooperative Reasoning +=

14. © Andrew IrelandDependable Systems Group Cooperative Reasoning += = Complementary techniques compensating for each other’s weaknesses

15. © Andrew IrelandDependable Systems Group Reliable & Safe Ada Subsets “It is not too late! I believe that by careful pruning of the Ada language, it is still possible to select a very powerful subset that would be reliable and efficient in implementation and safe and economic to use” C.A.R. Hoare 1980 ACM Turing Award Lecture

16. © Andrew IrelandDependable Systems Group The SPARK Approach A subset of Ada that eliminates potential ambiguities and insecurities (PRAXIS) Expressive enough for industrial applications, but restrictive enough to support rigorous analysis SPARK toolset supports data & information flow analysis and formal verification Partial correctness & exception freedom proof, e.g. proving code is free from buffer overflows, range violations, division by zero.

17. © Andrew IrelandDependable Systems Group SPARK Applications Advanced avionics (Eurofighter Typhoon), transportation, air traffic control, security (MONDEX) Ship Helicopter Operating Limits Information System (SHOLIS): first system developed to meet UK MOD Defence Standard 00-55 Supports “correctness-by-construction” and is advocated by US National Cyber Security Partnership (NSA) as one of three software development processes that can deliver sufficient assurance for security critical systems

18. © Andrew IrelandDependable Systems Group The SPARK Approach SPARK Examiner SPADE Simplifier SPADE Proof Checker VCs Cmds Unproven VCs  SPARK code Proofs

19. © Andrew IrelandDependable Systems Group NuSPADE Project SPARK Examiner SPADE Simplifier SPADE Proof Checker Bill J. Ellis (RA) EPSRC Critical Systems programme (GR/R24081) EPSRC RAIS Scheme (GR/T11289) http:// www.macs.hw.ac.uk/nuspade VCs Cmds Unproven VCs  Refinement Abstraction SPADEase SPARK code Proofs

20. © Andrew IrelandDependable Systems Group SPADEase Program Analysis Proof Planning Exception freedom proofs Automatic loop invariant discovery

21. © Andrew IrelandDependable Systems Group Conjecture Proof Planning Method Theory Proof Checker Tactic Proof Critic Proof Planner

22. © Andrew IrelandDependable Systems Group Proof Planning Reuse: strategies can be easily ported between checkers Robustness: critics and middle-out reasoning provide flexibility in how proof search is organized Cooperation: planning provides a natural level at which to combine multiple reasoning processes

23. © Andrew IrelandDependable Systems Group SPADEase Unproven VCs Cmds Abstract Predicates Annotations Cooperative reasoning, e.g. “productive use of failure” Proof Planner Program Analyzer

24. © Andrew IrelandDependable Systems Group subtype AR_T is Integer range 0..9; type A_T is array (AR_T) of Integer;... procedure Filter(A: in A_T; R: out Integer) is begin R:=0; for I in AR_T loop if A(I)>=0 and A(I)<=100 then R:=R+A(I); end if; end loop; end Filter; An Example

25. © Andrew IrelandDependable Systems Group H1: for_all (i___1: integer, ((i___1 >= ar_t__first) and (i___1 ((element(a, [i___1]) >= integer__first) and (element(a, [i___1]) <= integer__last))). H2: loop__1__i >= ar_t__first. H3: loop__1__i <= ar_t__last. H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. -> C1: r + element(a, [loop__1__i]) >= integer__first. C2: r + element(a, [loop__1__i]) <= integer__last. C3: loop__1__i >= ar_t__first. C4: loop__1__i <= ar_t__last.... H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. ->... C2: r + element(a,[loop__1__i]) <= integer__last.... Exception Freedom VC

26. © Andrew IrelandDependable Systems Group H1: for_all (i___1: integer, ((i___1 >= ar_t__first) and (i___1 ((element(a, [i___1]) >= integer__first) and (element(a, [i___1]) <= integer__last))). H2: loop__1__i >= ar_t__first. H3: loop__1__i <= ar_t__last. H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. -> C1: r + element(a, [loop__1__i]) >= integer__first. C2: r + element(a, [loop__1__i]) <= integer__last. C3: loop__1__i >= ar_t__first. C4: loop__1__i <= ar_t__last.... H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. ->... C2: r + element(a,[loop__1__i]) <= integer__last... Exception Freedom VC -32768. 32767.

27. © Andrew IrelandDependable Systems Group H1: for_all (i___1: integer, ((i___1 >= ar_t__first) and (i___1 ((element(a, [i___1]) >= integer__first) and (element(a, [i___1]) <= integer__last))). H2: loop__1__i >= ar_t__first. H3: loop__1__i <= ar_t__last. H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. -> C1: r + element(a, [loop__1__i]) >= integer__first. C2: r + element(a, [loop__1__i]) <= integer__last. C3: loop__1__i >= ar_t__first. C4: loop__1__i <= ar_t__last.... H4: element(a, [loop__1__i]) >= 0. H5: element(a, [loop__1__i]) <= 100. H6: r >= integer__first. H7: r <= integer__last. ->... C2: r + element(a,[loop__1__i]) <=... Exception Freedom VC -32768. 32767. If (32668 <= r <= 32767) then possible overflow, i.e. VC is unprovable 32767.

28. © Andrew IrelandDependable Systems Group Proof-Failure Analysis 1. Unproven relational goal of the form E Rel C, e.g. 2. Given the constraints on E, a counter-example can be found that shows bounds are insufficient to prove exception freedom, e.g. r + element(a,[loop__1__i]) <= 32767 r >= -32768 and r <= 32767 Patch: R >= ? and R <= ? Preconditions:

29. © Andrew IrelandDependable Systems Group SPADEase Unproven VCs Proof Planner Program Analyzer Annotations Abstract Predicates R >= ? and R <= ? subtype AR_T is Integer range 0..9; type A_T is array (AR_T) of Integer;... procedure Filter(A: in A_T; R: out Integer)is begin R:=0; for I in AR_T loop if A(I)>=0 and A(I)<=100 then R:=R+A(I); end if; end loop; end Filter;

30. © Andrew IrelandDependable Systems Group Program Analysis Solution: PURRS: Recurrence Relation Solver http://www.cs.unipr.it/purrs/

31. © Andrew IrelandDependable Systems Group Program Analysis

32. © Andrew IrelandDependable Systems Group Program Analysis Solutions:

33. © Andrew IrelandDependable Systems Group Combining Solutions --# assert R >= 0 and R <= I*100

34. © Andrew IrelandDependable Systems Group SPADEase Unproven VCs Proof Planner Program Analyzer Annotations --# assert R >= 0 and R <= I*100 Abstract Predicates R >= ? and R <= ? subtype AR_T is Integer range 0..9; type A_T is array (AR_T) of Integer;... procedure Filter(A: in A_T; R: out Integer)is begin R:=0; for I in AR_T loop if A(I)>=0 and A(I)<=100 then R:=R+A(I); end if; end loop; end Filter; subtype AR_T is Integer range 0..9; type A_T is array (AR_T) of Integer;... procedure Filter(A: in A_T; R: out Integer)is begin R:=0; for I in AR_T loop --# assert R >= 0 and R <= I*100 if A(I)>=0 and A(I)<=100 then R:=R+A(I); end if; end loop; end Filter;

35. © Andrew IrelandDependable Systems Group subtype AR_T is Integer range 0..9; type A_T is array (AR_T) of Integer;... procedure Filter(A: in A_T; R: out Integer) is begin R:=0; for I in AR_T loop --# assert R >= 0 and R <= I*100 if A(I)>=0 and A(I)<=100 then R:=R+A(I); end if; end loop; end Filter; Revised Filter Code

36. © Andrew IrelandDependable Systems Group Revised Exception Freedom VC H1: r >= 0. H2: r <= loop__1__i * 100. H3: for_all (i___1: integer, ((i___1 >= ar_t__first) and (i___1 ((element(a, [i___1]) >= integer__first) and (element(a, [i___1]) <= integer__last))). H4: loop__1__i >= ar_t__first. H5: loop__1__i <= ar_t__last. H6: element(a, [loop__1__i]) >= 0. H7: element(a, [loop__1__i]) <= 100. H8: r >= integer__first. H9: r <= integer__last. -> C1: r + element(a, [loop__1__i]) >= integer__first. C2: r + element(a, [loop__1__i]) <= integer__last. C3: loop__1__i >= ar_t__first. C4: loop__1__i <= ar_t__last. H1: r >= 0. H2: r <= loop__1__i * 100.... H6: element(a, [loop__1__i]) >= 0. H7: element(a, [loop__1__i]) <= 100. … ->... C2: r + element(a,[loop__1__i]) <= integer__last.... Transitivity proof plan 

37. © Andrew IrelandDependable Systems Group Loop Invariant VC H1: r >= 0. H2: r <= loop__1__i * 100.... H6: element(a, [loop__1__i]) >= 0. H7: element(a, [loop__1__i]) <= 100. … -> C1: r + element(a,[loop__1__i]) >= 0. C2: r + element(a,[loop__1__i]) <= (loop__1__i + 1) * 100. … Rippling proof plan 

38. © Andrew IrelandDependable Systems Group Given: Goal:

39. © Andrew IrelandDependable Systems Group P2_1 ✓ P2_2 ✓ P2_3 ✓ P2_4 ✓ P1_1 ✗✓✓ P1_2 ✗✓✓ P1_3 ✗✓✓ P1_4 ✗✓✓ P1_5 ✗✓✓ PhaseGoalProof Planning Proof-Failure Analysis Program Analysis P1 P2

40. © Andrew IrelandDependable Systems Group NuSPADE Results Our evaluation was based upon examples drawn from industrial data provided by Praxis, e.g. SHOLIS SPADE Simplifier is very effective on exception freedom VC, i.e. typical hit-rate of 92% SPADEase targeted the VCs which the SPADE Simplifier failed to prove, i.e. loop-based code While critical software is engineered to minimize the number and complexity of loops, we found that 80% of the loops we encountered were provable using SPADEase

41. © Andrew IrelandDependable Systems Group Program AnalysisProof Planning constraints not available post-VCGen soundness equational reasoning extensibility Compensating for Weaknesses?

42. © Andrew IrelandDependable Systems Group Limitations & Related Work Constraint solving fails when reasoning with “big numbers”, i.e. integers out with –(2 25 )… 2 25 -1 Precondition strengthening would improve our hit-rate, constraint solving may have a role to play. Integrating decision procedures with SPADEase would also improve hit-rate. Paul Jackson, Bill J. Ellis, Kathleen Sharp “Using SMT Solvers to Verify High-Integrity Programs” 2 nd Int. Workshop on Automated Formal Methods (AFM-07) Boulton, Bundy, Gordon, Slind Clam/HOL Project (`96-`99)

43. © Andrew IrelandDependable Systems Group Proof Plans making an Impact? Peter Amey Chief Technical Officer Praxis High Integrity Systems 2005 “SPADEase represents a very significant advance in the practical application of proof planning. …”

44. © Andrew IrelandDependable Systems Group Proof Plans making an Impact? Peter Amey Chief Technical Officer Praxis High Integrity Systems 2005 “… It increases the proportion of SPARK- generated verification conditions that can be proved automatically without introducing any new opaque, black-box processes. …”

45. © Andrew IrelandDependable Systems Group Proof Plans making an Impact? “… The separation of proof planning from proof checking also acts as a talent multiplier by allowing proof experts to spend their time creating new and reusable methods and approaches rather than working on individual proofs.” Peter Amey Chief Technical Officer Praxis High Integrity Systems 2005

46. © Andrew IrelandDependable Systems Group CORE Project Separation Logic: Reynolds (CMU) & O’Hearn (QMU) Focuses the reasoning effort on only those parts of the heap that are relevant to a program unit, i.e. local reasoning CORE Project: –Ewen Maclean (Post Doc) & Ianthe Hind (PhD) –EPSRC EP/F037597 –http://www.macs.hw.ac.uk/core Building upon shape analysis tools, CORE aims to automate the verification of functional properties

47. © Andrew IrelandDependable Systems Group Symbolic Execution (SmallfootRG) Proof Planning (IsaPlanner) shape properties content properties soundness extensibility Compensating for Weaknesses?

48. © Andrew IrelandDependable Systems Group CORE Challenges Automating “eureka” steps: Loop invariants Frame axioms Auxiliary inductive lemmas Existential witnesses Note: details via Tech Talks at Google

49. © Andrew IrelandDependable Systems Group Grand Challenges UK GC6: Dependable Systems Evolution (a collaboration with the International Verified Software Grand Challenge) Verifying Compiler Grand Challenge – Hoare The Verified Software Repository – Woodcock: accelerate the development of verification tools open access to challenge problems and results Industry engagement http://vsr.sourceforge.net/Introduction.htm

50. © Andrew IrelandDependable Systems Group Conclusion A new wave of software verification tools is emerging within a range of niche markets Technological advances and the economics of software failures have been major drivers Cooperation: tools and communities Grand Challenge or Emerging Technology? As Automated Reasoning researchers we are challenged by a grand opportunity!