Java Pathfinder JPF Tutorial - Test Input Generation With Java Pathfinder – Visser, Pasareanu, Khurshid Generalized Symbolic Execution for Model Checking and Testing – Khurshid, Pasareanu, Visser Symbolic Execution and Model Checking for Testing - Pasareanu, Visser JPF–SE - A symbolic execution extension to Java PathFinder – Anand, Pasareanu, Visser

JPF is an explicit state software model checker for Java bytecode

Usage As an exhaustive testing tool As a model checker Deploys abstractions and optimizations Extensibility – Search-/VMListeners – Model Java Interface (MJI) – Configurable ChoiceGenerators – Bytecode Factories (New, no tutorials yet)‏

What’s New Symbolic Test Data Generation The symbolic execution extension uses a BytecodeFactory to override JPF's core bytecodes to generate concrete test cases. In a nutshell, this works by using the JPF field/stackframe attribute system to collect symbolic path conditions, which are then fed into a contraint solver to obtain concrete test data. The choice points in this execution mode are branch instructions. Symbolic Test Data Generation Symbolic Threadsafety Test Generation This is a very simple "symbolic" execution mode that tries to identify potential thread safety problems, and then creates code to test this with concrete value JPF execution mode. The symbolic information is much more simple than with the symbc extension, and can be kept in the execution trace. The first implemented test category detects un-synchronized access of fields from within the same public (exported) method, to identify candidates for the PreciseRaceDetector listener. Symbolic Threadsafety Test Generation Compositional Verification Framework The CV extension implements a machine learning algorithm that can be used for assume/guarantee reasoning, to partition a system into components that can be verified separately. Ultimately, the goal of this extension is to dramatically improve the scalability of JPF, but it can be also used to generate environment assumptions for the UML model checking, to determine "valid" event sequences. This extension is mostly implemented with JPF listeners. Compositional Verification Framework Numeric Property Verification This bytecode replacement extension started as a set of numeric instruction classes to detect over/underflow, but got extended to also include inexact value propagation (NaN, Inf), exact floating point comparison, and potential catastrophic cancellation (loss of precision by subtraction/addition) Numeric Property Verification UML State Chart Model Checking The state chart framework is a variation of the user interface model checking, for the purpose of model checking UML state chart diagrams. It is a combination of MJI libraries that are based on a unique translation scheme for UML state charts, together with generic applications (test drivers) to execute these state charts. The framework supports both testing and model checking, using the same scripting language for environment specification. UML State Chart Model Checking

Symbolic Execution and Test-input Generation Willem Visser & Corina Pasareanu and Peter Mehlitz RIACS/Kestrel Technology/CSC NASA Ames Research Center

6 Overview Motivation for Model Checking Programs Introduction to Java PathFinder Symbolic Execution Coverage based Test-input generation Conclusions

7 Motivation Ariane 501 Mars Polar Lander Software Errors can be very costly

8 More Recently Software problem with Spirit

9 Model Checking OK Error trace or Finite-state model Temporal logic formula Model Checker  Line 5: … Line 12: … Line 15:… Line 21:… Line 25:… Line 27:… … Line 41:… Line 47:…

10 Overview Motivation for Model Checking Programs Introduction to Java PathFinder Symbolic Execution Coverage based Test-input generation Conclusions

11 Key Points Models can be infinite state –Unbounded objects, threads,… –Depth-first state generation (explicit-state)‏ –Verification requires abstraction Handle full Java language –mostly only for closed systems –Cannot handle native code no input/output through GUIs, files, Networks, … must be modeled by java code instead Allows Nondeterministic Environments –JPF traps special nondeterministic methods Checks for user-defined assertions, deadlock and LTL properties Incorporates a number of search strategies –DFS, BFS, A*, Best-first, etc.

12 Overview Motivation for Model Checking Programs Introduction to Java PathFinder Symbolic Execution Coverage based Test-input generation Conclusions

13 Concrete Execution Path (example)‏ x = 1, y = 0 1 >? 0 x = = 1 y = 1 – 0 = 1 x = 1 – 1 = 0 0 – 1 >? 0 int x, y; if (x > y) { x = x + y; y = x – y; x = x – y; if (x – y > 0)‏ assert(false); }

14 Symbolic Execution Tree (example)‏ x = X, y = Y X >? Y [ X > Y ] y = X + Y – Y = X [ X > Y ] x = X + Y – X = Y [ X > Y ] Y - X >? 0 [ X <= Y ] END[ X > Y ] x = X + Y [ X > Y, Y – X <= 0 ] END [ X > Y, Y – X > 0 ] END int x, y; if (x > y) { x = x + y; y = x – y; x = x – y; if (x – y > 0)‏ assert(false); }

15 Forward Symbolic Execution technique for executing a program on symbolic input values explore program paths –for each path, build a path condition –check satisfiability of path condition state –symbolic values of variables, path condition, and counter various applications –test generation –program verification traditional use –programs with fixed number of int variables

16 Challenges in Generalizing Symbolic Execution how to handle fields in dynamic structures? how to handle aliasing? how to generate tests? –satisfy criteria –satisfy precondition –are in-equivalent

17 Example class Node { int elem; Node next; Node swapNode() { if (next != null) if (elem > next.elem) { Node t = next; next = t.next; t.next = this; return t; } return this; } } ? null E0E1 E0 E1 null E0E1? E0E1 E0E1 Input list +ConstraintOutput list E0 > E1 none E0 <= E1 none E0 > E1 E1E0? E1E0 E1E0 E1E0 null E0E1 E0 ? null NullPointerException

18 Generalized Symbolic Execution model checker generates and explores “symbolic” execution tree –path conditions are added at each branch point off-the-shelf decision procedures check path conditions model checker backtracks if not satisfiable –non-determinism handles aliasing explore different heap configurations explicitly –concurrency lazy initialization –initializes program’s inputs on an “as-needed” basis –no a priori bound on input sizes preconditions to initialize inputs only with valid values

19 Algorithm (aliasing)‏ when method execution accesses field f if (f is uninitialized) { if (f is reference field of type T) { non-deterministically initialize f to  null  a new object of class T (with uninitialized fields)  an object created during prior field initialization (alias) } if (f is numeric/string field) initialize f to a new symbolic value }

20 Algorithm (illustration)‏ E0 next E1 next t null t E0 next E1 next ? E0 next E1 t next E0 next E1 next t E0 next E1 next t consider executing next = t.next; Precondition: acyclic list E0E1 next t null next t E0E1 next ?

21 program instrumentation counterexample(s)/test suite [heap+constraint+thread scheduling] Implementation via Instrumentation model checking decision procedure instrumented program correctness specification continue/ backtrack path condition (data)‏ heap configuration thread scheduling state: original program

22 Overview Motivation for Model Checking Programs Introduction to Java PathFinder Symbolic Execution Coverage based Test-input generation (TIG)‏ Conclusions

23 White- & Black-Box Testing void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } Requirements Specification Input Generator Oracle Testing Criteria Coverage of Specification & Code Testing Criteria Specification Coverage

24 Functional Spec White- & Black-Box Testing void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } Input Generator Oracle Testing Criteria Coverage of Specification & Code Testing Criteria Specification Coverage Input Spec Acyclic Linked List After removing the last element the list is empty Adding to a full list is not allowed

25 Model Checking & TIG OK Error trace or Executable Specification Property specifying coverage cannot be achieved Model Checker  Line 5: … Line 12: … Line 15:… Line 21:… Line 25:… Line 27:… … Line 41:… Line 47:… No test-input can achieve desired coverage Test-input to achieve coverage

26 Test-Input Generation (TIG) with Symbolic Execution … is it still true? White-box versus black-box –Symbolic execution most often white-box Simple data is straightforward Complex data –Black-box is (reasonably) straightforward – Korat (ISSTA’02)‏ –White-box? “… symbolic execution for testing programs is a more exploitable technique in the short term than the more general one of program verification” James King CACM 19:7, 1976

27 Functional Spec White- & Black-Box Testing for Complex Data void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } void add(Object o) { buffer[head] = o; head = (head+1)%size; } Object take() { … tail=(tail+1)%size; return buffer[tail]; } Input Generator Oracle Testing Criteria Coverage of Input Specification & Code Testing Criteria Input Specification Coverage Input Spec Class Invariant Pre-condition to every method boolean repOk()‏ no runtime errors exist

28 Red-Black Trees (1) The root is BLACK (2) Red nodes can only have black children (3) All paths from a node to its leaves contain the same number of black nodes. Self-balancing Binary Search Trees Java SortedMap Implementation (4) Acyclic (5) Consistent Parents repOk(): conditions (1)-(5)‏

29 repOk() Fragment boolean repOk(Entry e) { // root has no parent, root is black,… // RedHasOnlyBlackChildren workList = new LinkedList(); workList.add(e); while (!workList.isEmpty()) { Entry current=(Entry)workList.removeFirst(); Entry cl = current.left; Entry cr = current.right; if (current.color == RED) { if(cl != null && cl.color == RED) return false; if(cr != null && cr.color == RED) return false; } if (cl != null) workList.add(cl); if (cr != null) workList.add(cr); } // equal number of black nodes on left and right sub-tree… return true; }

30 Black-box TIG Generate inputs based on analysis of input structure –e.g. Rover plan generation, Korat 100% “coverage” of input structures up to a predetermined upper-bound –e.g. all red-black trees with 6 or less nodes Complex data requires that only valid structures be considered –Use class invariant to reduce number of input structures to consider a predicate characterizing all the instances of a class boolean repOk() Check code coverage using generated structures as input Advantage – test code for which no source is available

31 Symbolic execution for black-box TIG Symbolic execution of repOk()‏ –Generate new structures only when repOk() returns true –Limit the size of the structures generated –Only correct structures will be generated repOk() returns true after all nodes in the tree have been visited, hence they must all be concrete symbolic (partial) structures can fail repOk()‏ Similar to Korat –Except we can also deal with data constraints

32 Symbolic Execution of repOk() Example public static boolean repOk(Entry e) { if (e == null)‏ return true; if (e.color == RED)‏ return false; …

33 White-box TIG Consider code coverage criterion when generating test inputs Challenge –Treating complex data with symbolic execution Use repOk() as a method precondition during symbolic execution of source code: –use repOk() to convert “symbolic” input structures into concrete structures that cover the code and pass repOk()‏ –use repOk() also to eliminate “symbolic” structures during lazy initialization, thus reducing the search space

34 repOk() x 2 abstract and concrete Symbolic Execution of Code During Lazy Initialization check Abstract repOK()‏ When coverage is achieved, solve the symbolic constraints to create concrete inputs To concretize inputs by symbolic execution of Concrete repOk()‏ over symbolic structures - as with Black-box TIG -

35 White-box TIG: cover branches in deleteEntry(Entry p)‏ /* precondition: p.repOk() */ private void deleteEntry(Entry p) { if (p.left != null && p.right != null) { Entry s = successor(p); swapPosition(s, p); } Entry replacement = (p.left != null ? p.left : p.right); if (replacement != null) { replacement.parent = p.parent; if (p.parent == null)‏ root = replacement; else if (p == p.parent.left) { p.parent.left = replacement; } else p.parent.right = replacement; p.left = p.right = p.parent = null; if (p.color == BLACK)‏ fixAfterDeletion(replacement);...

36 Symbolic Execution for white-box TIG if (p.left != null && p.right != null) {... Symbolic structure before executing branch Concretize Concrete structure that will cover the code The symbolic structure is used as input to repOk() and lazily executed to obtain the concrete structure Symbolic structure(s)‏ that cover the branch This structure “passes” the abstract repOk()‏

37 Conservative repOk()‏ Used to eliminate symbolic structures that cannot be converted to a concrete structure that satisfy repOk() and therefore do not describe valid inputs Because of symbolic structures we use abstraction –conservative_RepOk() can return TRUE, FALSE or Don’t Know if FALSE, ignore that structure by backtracking if TRUE or Don’t Know, continue... Example: (2) Red nodes have only black children. FALSE TRUEDon’t Know

38 Conservative repOk()‏ // root has no parent, root is black,… // RedHasOnlyBlackChildren workList = new LinkedList(); workList.add(e); while (!workList.isEmpty()) { Entry current=(Entry)workList.removeFirst(); Entry cl = current.left; Entry cr = current.right; if (current.color == RED) { if ( current._left_is_initialized && cl != null && cl.color == RED) return false; if(current._right_is_initialized && cr != null && cr.color == RED) return false; } if (current._left_is_initialized && cl != null) workList.add(cl); if (current._right_is_initialized && cr != null) workList.add(cr); } // equal number of black nodes on left and right sub-tree… return true;

39 Cover branches in deleteEntry(Entry p)‏ /* precondition: p.repOk() */ private void deleteEntry(Entry p) { if (p.left != null && p.right != null) { Entry s = successor(p); swapPosition(s, p); } Entry replacement = (p.left != null ? p.left : p.right); if (replacement != null) { replacement.parent = p.parent; if (p.parent == null)‏ root = replacement; else if (p == p.parent.left) { p.parent.left = replacement; } else p.parent.right = replacement; p.left = p.right = p.parent = null; if (p.color == BLACK)‏ fixAfterDeletion(replacement);...

40 Lazy Initialization from Partial Structures Partial structure satisfying conservative_RepOk()‏ Solution that satisfies repOk()‏Not a solution! Concretization By lazy initialization of repOK()‏

41 Black Box Results Tests (16)‏ (6)‏ (4)‏ N fixD %BC (2)‏ (2)‏ (1)‏3 fixIns %BC delEnt %BC Candidate Structures StructsTime Size 7: Korat candidates vs 35804

42 White-box 11(53)‏ Tests 100 fixD %BC fixIns %BC delEnt %BC Candidate Structures MemTime

43 Conclusions Other JPF features –Partial-order reductions –Observations Test-input Generation –Examples with primitive data as well as complex data –Make link with Shape Analysis Derive conservative repOk() from concrete repOk() automatically Symbolic Execution –Invariant generation Combining Test-input generation and runtime monitoring –X9 testing framework for a next generation Mars Rover