Extended Static Checking for Java Cormac Flanagan K. Rustan M. Leino Mark Lillibridge Greg Nelson James B. Saxe Raymie Stata Compaq SRC 18 June 2002 PLDI02, Berlin, Germany

ESC/Java Vision Increased programmer productivity and program reliability through increased rigor Increased programmer productivity and program reliability through increased rigor Record design decisions +Utilize automatic checking =Detect errors and improve maintainability

ESC/Java Users view ESC/Java Annotated Java program Warning messages public class Bag { private int[] a; private int n; invariant 0 <= n && n <= a.length; public int[] initialElements) { n = initialElements.length; a = new int[n]; System.arraycopy(initialElements, 0, a, 0, n); } public void add(int x) { if (n == a.length) { int[] b = new int[2*(a.length+1)]; System.arraycopy(a, 0, b, 0, n); a = b; } a[n] = x; n++; } public int extractMin() { int m = Integer.MAX_VALUE; int mindex = 0; for (int i = 0; i < n; i++) { if (a[i] < m) { mindex = i; m = a[i]; } } if (0 < n) { n--; a[mindex] = a[n]; } return m; } // The program text continues down here, but if youre // reading this, you probably arent paying attention to // the talk. Bag.java:18: Array index possibly too large

ESC/Java ESC/Java distinguishing features Modular checking Modular checking Annotation language captures design decisions Annotation language captures design decisions Powered by automatic theorem prover Powered by automatic theorem prover Not decidable Not decidable

ESC/Java Demo

Design tradeoffs Missed errors Missed errors Spurious warnings Spurious warnings Annotation overhead Annotation overhead Performance Performance

ESC/Java Tool architecture Translator Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Valid Resource exhausted

ESC/Java Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Tool architecture, detail Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Simple Simple non_null non_null Method annotations Method annotations requires E; requires E; modifies w; modifies w; ensures P; ensures P; exsures (T x) Q; exsures (T x) Q; Object invariants Object invariants invariant E; invariant E;

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Simple Simple non_null non_null Method annotations Method annotations requires E; requires E; modifies w; modifies w; ensures P; ensures P; exsures (T x) Q; exsures (T x) Q; Object invariants Object invariants invariant E; invariant E;

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Specification expressions Specification expressions side-effect free Java expressions side-effect free Java expressions no ++, no method calls no ++, no method calls \result, \old(E) \result, \old(E) ensures \result == \old(x); ensures \result == \old(x); ==> ==> (\forall T x; P), (\exists T x; P) (\forall T x; P), (\exists T x; P) (\forall int j; 0 a[j] > 0); (\forall int j; 0 a[j] > 0); \typeof(E), \type(T), <: \typeof(E), \type(T), <: requires \typeof(x) == \typeof(this); requires \typeof(x) == \typeof(this);

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Specification expressions Specification expressions side-effect free Java expressions side-effect free Java expressions no ++, no method calls no ++, no method calls \result, \old(E) \result, \old(E) ensures \result == \old(x); ensures \result == \old(x); ==> ==> (\forall T x; P), (\exists T x; P) (\forall T x; P), (\exists T x; P) (\forall int j; 0 a[j] > 0); (\forall int j; 0 a[j] > 0); \typeof(E), \type(T), <: \typeof(E), \type(T), <: requires \typeof(x) == \typeof(this); requires \typeof(x) == \typeof(this);

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Concurrency Concurrency monitored_by lock monitored_by lock monitored_by this */long x; monitored_by this */long x; \lockset[lock] \lockset[lock] requires \lockset[this]; requires \lockset[this]; lock0 < lock1 lock0 < lock1 \max(\lockset) \max(\lockset) requires \max(\lockset) < this; requires \max(\lockset) < this;

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Concurrency Concurrency monitored_by E monitored_by E monitored_by this */long x; monitored_by this */long x; \lockset[lock] \lockset[lock] requires \lockset[this]; requires \lockset[this]; lock0 < lock1 lock0 < lock1 \max(\lockset) \max(\lockset) requires \max(\lockset) < this; requires \max(\lockset) < this;

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Ghost variables Ghost variables ghost public T x; ghost public T x; ghost public int objectState; ghost public int objectState; ghost public \TYPE elementType; ghost public \TYPE elementType; set x = E; set x = E; set objectState = Open; set objectState = Open; set elementType = \type(T); set elementType = \type(T);

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Ghost variables Ghost variables ghost public T x; ghost public T x; ghost public int objectState; ghost public int objectState; ghost public \TYPE elementType; ghost public \TYPE elementType; set x = E; set x = E; set objectState = Open; set objectState = Open; set elementType = \type(T); set elementType = \type(T);

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Annotation language Miscellaneous Miscellaneous assert E; assert E; assume E; assume E; assume x >= 0; // because x == y*y assume x >= 0; // because x == y*y nowarn nowarn x = a[j]; nowarn x = a[j]; nowarn axiom E; axiom E; axiom (\forall int x; x >> 2 >= 0); axiom (\forall int x; x >> 2 >= 0);

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands S,T::=assert E S,T::=assert E |assume E |x = E |raise |S ; T |S ! T |S [] T |loop {inv E} S T end |call x = t.m(E) |…

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands x = t.f.g; x = t.f.g; assert t != null; tmp = select(f, t); assert tmp != null; x = select(g, tmp) if (x = 0; */ if (x = 0; */ (assume x < 0; x = -x []assume !(x < 0) ); assert x >= 0

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Sugared commands x = t.f.g; x = t.f.g; assert t != null); tmp = select(f, t)); assert tmp != null); x = select(g, tmp) if (x = 0; */ if (x = 0; */ (assume x < 0; assume lblpos(Then^280:7, true); x = -x []assume !(x < 0); assume lblpos(Else^280:7, true) ); assert x >= 0

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Primitive commands S,T::=assert E |assume E |x = E |raise |S ; T |S ! T |S [] T |loop {inv E} S T end |call x = t.m(E) |…

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Primitive commands requires Pre; modifies w; ensures Post; void m(U u); requires Pre; modifies w; ensures Post; void m(U u); call x = t.m(E) call x = t.m(E) var u in u = E; assert Pre; var w 0 in w 0 = w; havoc w; assume Post endend

ESC/Java |raise |S ; T |S ! T |S [] T Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Passive commands S,T::=assert E |assume E |x = E

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Passive commands if (x = 0; */ if (x = 0; */ (assume x 0 < 0; assume x 1 == -x 0 ; assume x 2 == x 1 []assume !(x 0 < 0); assume x 2 == x 0 ); assert x 2 >= 0

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Weakest preconditions wp(assert E, Q) = E && Q wp(assert E, Q) = E && Q wp(assume E, Q) = E ==> Q wp(assume E, Q) = E ==> Q wp(S;T, Q) = wp(S, wp(T,Q)) wp(S;T, Q) = wp(S, wp(T,Q)) wp(S [] T, Q) = wp(S, Q) && wp(T, Q) wp(S [] T, Q) = wp(S, Q) && wp(T, Q) wp(S, Q) = wp(S, true) && wlp(S, Q) wp(S, Q) = wp(S, true) && wlp(S, Q) wlp(S, Q) = wlp(S, false) || Q wlp(S, Q) = wlp(S, false) || Q

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition Universal background predicate Universal background predicate (FORALL (t) (<: t t)) (FORALL (t) (<: t t)) Type-specific background predicate Type-specific background predicate (<: T_T |T_java.lang.Object|) (<: T_T |T_java.lang.Object|) Verification condition: BP Univ && BP T ==> VC method Verification condition: BP Univ && BP T ==> VC method

ESC/Java (BG_PUSH (AND (<: T_T |T_java.lang.Object|) (EQ T_T (asChild T_T |T_java.lang.Object|)) (DISTINCT arrayType |T_boolean| |T_char| |T_byte| |T_short| |T_int| |T_long| |T_float| |T_double| |T_.TYPE| T_T |T_java.lang.Object|))) (EXPLIES (LBLNEG |vc.T.abs.2.2| (IMPLIES (AND (EQ elems) (EQ elems (asElems elems)) (< (eClosedTime elems) alloc) (EQ LS (asLockSet LS)) (EQ alloc)) (NOT (AND (EQ (is |x:2.21| T_int)) (OR (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ (EQ |x:2.21 | |x:2.21|))) (NOT (LBLNEG (>= |x:2.21 | 0)))) (AND (OR (AND (< |x:2.21| 0) (LBLPOS |trace.Then^0,3.15| (EQ (EQ |x:3.17| (- 0 |x:2.21|)) (EQ |x:2.21 | |x:3.17|)) (AND (NOT (< |x:2.21| 0)) (LBLPOS |trace.Else^1,3.4| (EQ (EQ |x:2.21 | |x:2.21|))) (LBLNEG (>= |x:2.21 | 0)) (NOT (LBLNEG (EQ |ecReturn| |ecReturn|))))))))) (AND (DISTINCT |ecReturn|))) Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Verification condition class T { class T { static int abs(int x) { static int abs(int x) { if (x < 0) { x = -x; } if (x < 0) { x = -x; } assert x >= 0; assert x >= 0; }}

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Theorem prover: Simplify Nelson-Oppen cooperating decision procedures Nelson-Oppen cooperating decision procedures conguence closure conguence closure linear arithmetic linear arithmetic partial orders partial orders quantifiers quantifiers Key features: Key features: automatic: no user interaction automatic: no user interaction refutation based: searches for counterexamples refutation based: searches for counterexamples heuristics tuned for program checking heuristics tuned for program checking labels labels time limit time limit

ESC/Java Annotated Java program Verification condition Counterexample context Warning messages Automatic theorem prover Post processor Sugared command Primitive command Passive command Translator Counterexamples and warnings Counterexample: labels: |vc.Bag.add.20.2| |trace.Then^0,21.23|) context: (AND (NEQ |tmp1!a:23.23| null) (NEQ this null) (EQ alloc) (EQ |tmp4!n:26.6| 0) … (<= alloc (vAllocTime |tmp3!a:26.4|)) ) Bag: add(int) Bag.java:26: Warning: Array index possibly too large (IndexTooBig) a[n] = x; ^ Execution trace information: Executed then branch in "Bag.java", line 21, col

ESC/Java Experience Javafe (ESC/Java front end, ~40kloc)~12 Javafe (ESC/Java front end, ~40kloc)~12 Mercator (web crawler)2 Mercator (web crawler)2 Sparse matrix library2 Sparse matrix library2 Annotation wizard2 Annotation wizard2 Performance plotting program1 Performance plotting program1 …… …… External user experience… External user experience… Experience with automatically inferred annotations Experience with automatically inferred annotations Errors found

ESC/Java Performance 50% of all methods:< 0.5 s 50% of all methods:< 0.5 s 80% of all methods:< 1 s 80% of all methods:< 1 s time limit:300 s time limit:300 s total time for Javafe (~40kloc): 65 min. total time for Javafe (~40kloc): 65 min.

ESC/Java Experience: annotations Capture common design decisions Capture common design decisions Suggested immediately by warnings Suggested immediately by warnings Overhead: 4-10% of source code Overhead: 4-10% of source code ~1 annotation per field or parameter ~1 annotation per field or parameter Most common annotations: Most common annotations: non_null non_null container element types container element types

ESC/Java Related work ESC/Modula-3 ESC/Modula-3 [Detlefs, Leino, Nelson, Saxe; ] [Detlefs, Leino, Nelson, Saxe; ] Full functional Java specification and verification Full functional Java specification and verification JML, LOOP JML, LOOP Languages and language features Languages and language features Euclid, Eiffel, B, Escher, Guava, Vault, Cqual, … Euclid, Eiffel, B, Escher, Guava, Vault, Cqual, … LCLint, refinement types, Types against races LCLint, refinement types, Types against races Other checking techniques Other checking techniques Abstract interpretation, PREfix, SLAM, Bandera, Java PathFinder 2, Canvas, ESP, AST Toolkit, Metal, … Abstract interpretation, PREfix, SLAM, Bandera, Java PathFinder 2, Canvas, ESP, AST Toolkit, Metal, … Tools built on ESC/Java Tools built on ESC/Java Daikon, Houdini Daikon, Houdini

ESC/Java Conclusions Theorem proving for program analysis works! Theorem proving for program analysis works! Annotations express design decisions Annotations express design decisions Cost effective? Cost effective? Ready for use by disciplined programming teams and in teaching Ready for use by disciplined programming teams and in teaching Future challenges: Future challenges: beyond method-modular checking beyond method-modular checking more specification methodologies more specification methodologies Download ESC/Java: research.compaq.com/SRC/esc Sources now available, too! Download ESC/Java: research.compaq.com/SRC/esc Sources now available, too!