Extended Static Checking for Java or Light-weight formal methods: from objects to components Joint work with Cormac Flanagan, Mark Lillibridge, Greg Nelson, James B. Saxe, Raymie Stata Compaq SRC 6 Nov 2002 FMCO 2002, Leiden, The Netherlands K. Rustan M. Leino Microsoft Research, Redmond, WA

Formal methods coverage effort Note: Illustration not to scale type checking program verification decidability ceiling extended static checking light-weight formal methods

User’s view ESC/Java Annotated Java program Warning messages public class Bag { private int[] a; private int n; invariant 0 <= n && n <= a.length; public int[] initialElements) { n = initialElements.length; a = new int[n]; System.arraycopy(initialElements, 0, a, 0, n); } public void add(int x) { if (n == a.length) { int[] b = new int[2*(a.length+1)]; System.arraycopy(a, 0, b, 0, n); a = b; } a[n] = x; n++; } public int extractMin() { int m = Integer.MAX_VALUE; int mindex = 0; for (int i = 0; i < n; i++) { if (a[i] < m) { mindex = i; m = a[i]; } } if (0 < n) { n--; a[mindex] = a[n]; } return m; } // The program text continues down here, but if you’re // reading this, you probably aren’t paying attention to // the talk. Bag.java:18: Array index possibly too large Bag.java:45: Precondition possibly violated

Goals of ESC/Java  Practical static checking  Detect common run-time errors null dereferences null dereferences array bounds array bounds type casts type casts race conditions race conditions deadlocks deadlocks  Modular checking

ESC/Java distinguishing features Annotation language captures design decisions Annotation language captures design decisions Powered by automatic theorem prover Powered by automatic theorem prover Not decidable Not decidable Not sound or complete Not sound or complete Performs modular checking Performs modular checking

Method-modular checking Check that each method satisfies its specification, assuming that all called routines satisfy theirs Check that each method satisfies its specification, assuming that all called routines satisfy theirs Reason about implementation when it is written, not when it is used or extended Reason about implementation when it is written, not when it is used or extended

Modular checking Method body Client Interface check check Client check

Design tradeoffs Missed errors Missed errors Spurious warnings Spurious warnings Annotation overhead Annotation overhead Performance Performance

Tool architecture Translator Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Valid Resource exhausted

Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Simple Simple non_null non_null Method annotations Method annotations requires E; requires E; modifies w; modifies w; ensures P; ensures P; exsures (T x) Q; exsures (T x) Q; Object invariants Object invariants invariant E; invariant E;

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Specification expressions Specification expressions side-effect free Java expressions side-effect free Java expressions no ++, no method calls no ++, no method calls \result, \old(E) \result, \old(E) ensures \result == \old(x); ensures \result == \old(x); ==> ==> (\forall T x; P), (\exists T x; P) (\forall T x; P), (\exists T x; P) (\forall int j; 0 a[j] > 0); (\forall int j; 0 a[j] > 0); \typeof(E), \type(T), <: \typeof(E), \type(T), <: requires \typeof(x) == \typeof(this); requires \typeof(x) == \typeof(this);

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Concurrency Concurrency monitored_by lock monitored_by lock monitored_by this */long x; monitored_by this */long x; \lockset[lock] \lockset[lock] requires \lockset[this]; requires \lockset[this]; lock0 < lock1 lock0 < lock1 \max(\lockset) \max(\lockset) requires \max(\lockset) < this; requires \max(\lockset) < this;

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Concurrency Concurrency monitored_by lock monitored_by lock monitored_by this */long x; monitored_by this */long x; \lockset[lock] \lockset[lock] requires \lockset[this]; requires \lockset[this]; lock0 < lock1 lock0 < lock1 \max(\lockset) \max(\lockset) requires \max(\lockset) < this; requires \max(\lockset) < this;

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Ghost variables Ghost variables ghost public T x; ghost public T x; ghost public int objectState; ghost public int objectState; ghost public \TYPE elementType; ghost public \TYPE elementType; set x = E; set x = E; set objectState = Open; set objectState = Open; set elementType = \type(T); set elementType = \type(T);

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Ghost variables Ghost variables ghost public T x; ghost public T x; ghost public int objectState; ghost public int objectState; ghost public \TYPE elementType; ghost public \TYPE elementType; set x = E; set x = E; set objectState = Open; set objectState = Open; set elementType = \type(T); set elementType = \type(T);

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Miscellaneous Miscellaneous assert E; assert E; assume E; assume E; assume x >= 0; // because x == y*y assume x >= 0; // because x == y*y nowarn nowarn x = a[j]; nowarn x = a[j]; nowarn axiom E; axiom E; axiom (\forall int x; x >> 2 >= 0); axiom (\forall int x; x >> 2 >= 0);

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Weakest preconditions wp(assert E, Q) = E  Q wp(assert E, Q) = E  Q wp(assume E, Q) = E  Q wp(assume E, Q) = E  Q wp(S;T, Q) = wp(S, wp(T,Q)) wp(S;T, Q) = wp(S, wp(T,Q)) wp(S [] T, Q) = wp(S, Q)  wp(T, Q) wp(S [] T, Q) = wp(S, Q)  wp(T, Q) wp(S, Q) = wp(S, true)  wlp(S, Q) wp(S, Q) = wp(S, true)  wlp(S, Q) wlp(S, Q) = wlp(S, false)  Q wlp(S, Q) = wlp(S, false)  Q

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition Universal background predicate Universal background predicate (FORALL (t) (<: t t)) (FORALL (t) (<: t t)) Type-specific background predicate Type-specific background predicate (<: T_T |T_java.lang.Object|) (<: T_T |T_java.lang.Object|) Verification condition: BP Univ  BP T  VC method Verification condition: BP Univ  BP T  VC method

Verification condition generation  Easy for small languages [Dijkstra]  Much harder for real languages  Object-oriented  Typed  Dynamic allocation  Exceptions  Aliasing  Threads

Verification conditions for real programs Java Guarded command Verification condition x = a[ i++ ]; i0 = i; i = i + 1; assert (LABEL a != null); assert (LABEL 0 <= i0); assert (LABEL i0 < a.length); x = elems[a][i0];  i0.(i0 == i ==> … ) wlp assume preconditions assume invariants assert postconditions assert invariants

Verification condition  Formula in untyped, first-order predicate calculus equality and function symbols equality and function symbols quantifiers quantifiers arithmetic operations arithmetic operations select and store operations select and store operations Eg.  x.  y.(x > y ==> … ) Eg.  x.  y.(x > y ==> … )

Background axioms  Additional properties of Java that the theorem prover needs to know  A variable of type T always holds a value whose type is a subtype of T  The subtyping relation is reflexive, anti-symmetric, and transitive  new returns an object that is distinct from all existing objects ... lots more...  java.lang.Object has no supertype

Example verification condition  Verification condition large but “dumb” (IMPLIES (DISTINCT |ecReturn| |L_14.4|) (IMPLIES (AND (EQ |a:2.8|) (EQ |a:2.8| (asField |a:2.8| (array |T_int|))) ( | |T_Bag|)) (EQ (isAllocated |this | alloc)) (NEQ |this | null)) (FORALL (tmp1 |tmp2:21.4| |tmp3:21.6| |m:12.8| |mindex:13.8| |i:14.13| |tmp0:14.28|) (AND (IMPLIES ( |)) (AND (LBLNEG (NEQ (select |a:2.8| |this |) null)) (LBLNEG ( |)))) (IMPLIES ( |)) 1) |MAX_VALUE:3.4.26|) (AND (LBLNEG (NEQ (select |a:2.8| |this |) null)) (LBLNEG ( |)))) (FORALL (|m:17.8|) (IMPLIES (EQ |m:17.8| (select (select elems (select |a:2.8| |this |)) 1)) (FORALL (|i:14.28|) (IMPLIES (AND (EQ |i:14.28| (+ 1 1)) (EQ |bool$false|)) (FORALL (|tmp2:21.4 |) (IMPLIES (EQ |tmp2:21.4 | (select |a:2.8| |this |)) (AND (LBLNEG (NEQ (select |a:2.8| |this |) null)) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |))) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |) (arrayLength (select |a:2.8| |this |)))) (LBLNEG (NEQ |tmp2:21.4 | null)) (LBLNEG ( |))) (LBLNEG (EQ |ecReturn| |ecReturn|))))))))))) (IMPLIES (NOT ( |)) 1) |MAX_VALUE:3.4.26|)) (FORALL (|i:14.28|) (IMPLIES (AND (EQ |i:14.28| (+ 1 1)) (EQ |bool$false|)) (FORALL (|tmp2:21.4 |) (IMPLIES (EQ |tmp2:21.4 | (select |a:2.8| |this |)) (AND (LBLNEG (NEQ (select |a:2.8| |this |) null)) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |))) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |) (arrayLength (select |a:2.8| |this |)))) (LBLNEG (NEQ |tmp2:21.4 | null)) (LBLNEG ( |))) (LBLNEG (EQ |ecReturn| |ecReturn|)))))))))) (IMPLIES (NOT ( |))) (AND (IMPLIES (EQ |L_14.4| |L_14.4|) (FORALL (|tmp2:21.4 |) (IMPLIES (EQ |tmp2:21.4 | (select |a:2.8| |this |)) (AND (LBLNEG (NEQ (select |a:2.8| |this |) null)) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |))) (LBLNEG ( | (- (select |n:3.6| |this |) 1)) |this |) (arrayLength (select |a:2.8| |this |)))) (LBLNEG (NEQ |tmp2:21.4 | null)) (LBLNEG ( |))) (LBLNEG (EQ |ecReturn| |ecReturn|)))))) (IMPLIES (NOT (EQ |L_14.4| |L_14.4|)) (AND (LBLNEG (EQ |L_14.4| |ecReturn|))))))))))

(BG_PUSH (AND (<: T_T |T_java.lang.Object|) (EQ T_T (asChild T_T |T_java.lang.Object|)) (DISTINCT arrayType |T_boolean| |T_char| |T_byte| |T_short| |T_int| |T_long| |T_float| |T_double| |T_.TYPE| T_T |T_java.lang.Object|))) (EXPLIES (LBLNEG |vc.T.abs.2.2| (IMPLIES (AND (EQ elems) (EQ elems (asElems elems)) (< (eClosedTime elems) alloc) (EQ LS (asLockSet LS)) (EQ alloc)) (NOT (AND (EQ (is |x:2.21| T_int)) (OR (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ (EQ |x:2.21 | |x:2.21|))) (NOT (LBLNEG (>= |x:2.21 | 0)))) (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ (EQ |x:2.21 | |x:2.21|))) (LBLNEG (>= |x:2.21 | 0)) (NOT (LBLNEG (EQ |ecReturn| |ecReturn|))))))))) (AND (DISTINCT |ecReturn|))) Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition class T { class T { static int abs(int x) { static int abs(int x) { if (x < 0) { x = -x; } if (x < 0) { x = -x; } assert x >= 0; assert x >= 0; }}

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Theorem prover: “Simplify” Nelson-Oppen cooperating decision procedures Nelson-Oppen cooperating decision procedures congruence closure congruence closure linear arithmetic linear arithmetic partial orders partial orders quantifiers quantifiers Key features: Key features: automatic: no user interaction automatic: no user interaction refutation based: searches for counterexamples refutation based: searches for counterexamples heuristics tuned for program checking heuristics tuned for program checking labels labels time limit time limit

Automatic theorem proving Verification condition Automatic theorem prover (Simplify) Counterexample  x.  y.(x > y ==> … ) DivergesValid

Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Counterexamples and warnings Counterexample: labels: |vc.Bag.add.20.2| |trace.Then^0,21.23|) context: (AND (NEQ |tmp1!a:23.23| null) (NEQ this null) (EQ alloc) (EQ |tmp4!n:26.6| 0) … (<= alloc (vAllocTime |tmp3!a:26.4|)) ) Bag: add(int) Bag.java:26: Warning: Array index possibly too large (IndexTooBig) a[n] = x; ^ Execution trace information: Executed then branch in "Bag.java", line 21, col

ESC/Java research platform Annotation inference Annotation inference Daikon [Ernst], Houdini [Flanagan & Leino] Daikon [Ernst], Houdini [Flanagan & Leino] Other checking tools Other checking tools Calvin [Qadeer et al.], Stale-value concurrency checker [Burrows & Leino] Calvin [Qadeer et al.], Stale-value concurrency checker [Burrows & Leino] Formal verification without driving theorem prover Formal verification without driving theorem prover Simplify theorem prover Simplify theorem prover SLAM [Ball, Rajamani, et al.], oolong [Leino et al.], … SLAM [Ball, Rajamani, et al.], oolong [Leino et al.], … Teaching Teaching Kansas State University [Dwyer & Hatcliff] Kansas State University [Dwyer & Hatcliff]