1. 1/23/2003University of Virginia1 Korat: Automated Testing Based on Java Predicates CS751 Presentation by Radu Stoleru C.Boyapaty, S.Khurshid, D.Marinov

2. 1/23/2003University of Virginia2 Roadmap Why do they do it? Statement of the Problem State of the Art What do they do? How do they do it? Test Input Generation Checking Correctness Results & Evaluation Questions & Comments

3. 1/23/2003University of Virginia3 Roadmap

4. 1/23/2003University of Virginia4 Why do they do it? Automated Testing. Why improve testing? Through manual testing: significant errors are not found it takes 30% of development time automated testing is an industry standard validation Automated Testing consists of: automated generation of test cases from specifications automated execution of test cases automated validation

5. 1/23/2003University of Virginia5 Why do they do it? Specification-based testing: Z specification, UML statechart – no linked data structs TestEra framework (’01) – new specification language JML+JUnit (’01) – no test case generation Static Analysis Extended Static Checker (’98) – no complex structs TVLA (’98) – only limited program properties Software model checking JavaPathFinder (’00), VeriSoft (’98) – no linked data structs

6. 1/23/2003University of Virginia6 Why do they do it? Korat: automated generation of test cases for complex structs complete evaluation of correctness automatically generates counter-examples no new specification language Testing Framework JUnitJML+JUnitKorat generating test cases generating test oracle running tests

7. 1/23/2003University of Virginia7 Roadmap

8. 1/23/2003University of Virginia8 What do they do? use JML for formal specification (class invariants, preconditions, postconditions) generate test inputs using preconditions builds Java predicate builds a skeleton finitization prunes input state space generates isomorph-free test cases evaluate correctness using postconditions using JML/JUnit

9. 1/23/2003University of Virginia9 Roadmap

10. 1/23/2003University of Virginia10 Example class BinaryTree { //@ public invariant //@ repOk(); Node root; int size; static class Node { Node left; Node right; } /*@ public normal_behavior @ requires has(n); @ ensures !has(n); @*/ void remove(Node n) {... } } boolean repOk() { if (root == null) return size == 0; Set visited = new HashSet(); visited.add(root); List workList = new LinkedList(); workList.add(root); while (!workList.isEmpty()) { Node current = (Node)workList.removeFirst(); if (current.left != null) { if (!visited.add(current.left)) return false; workList.add(current.left); } if (current.right != null) { if (!visited.add(current.right)) return false; workList.add(current.right); } if (visited.size() != size) return false; return true;}

11. 1/23/2003University of Virginia11 Input Size 5 non-isomorphic solutions for 3 nodes: N0N0 N1N1 N2N2 N0N0 N1N1 N2N2 (n+1) 2n+1 candidates for n nodes (2 92 for 12 nodes) how to find them quickly? left right N1N1 N2N2 left right N0N0 N1N1 N2N2 left N0N0 N1N1 N2N2 right N0N0

12. 1/23/2003University of Virginia12 Search Korat search algorithm: void koratSearch(Predicate p, Finitization f) { initialize(f); while(hasNextCandidate()) { Object candidate = nextCandidate(); try { if(p.invoke(candidate)) output(candidate); } catch (Throwable t) {} backtrack(); } given a predicate and a finitization, candidate inputs are generated inputs are validated by invoking the predicate on them

13. 1/23/2003University of Virginia13 Finitization a set of bounds that limits the size of the input Class Domain := a set of objects from one class {N 0, N 1, N 2 } Field Domain := a set of values a field can take. For Node.left it is {null, N 0, N 1, N 2 } Finitization finBinaryTree(int n, int min, int max) { Finitization f = new Finitization(BinaryTree.class); ObjSet nodes = f.createObjects(“Node”, n); nodes.add(null); f.set(“root”, nodes); // Field Domain f.set(“size”, new IntSet(min, max)); // Field Domain f.set(“Node.left”, nodes); // Field Domain f.set(“Node.right”, nodes); // Field Domain return f; } generated automatically by Korat can be further specialized

14. 1/23/2003University of Virginia14 State Space using a finitization, Korat: allocates a given number of objects constructs candidate vectors using object fields: ‘root’, ‘left’, ‘right’: {null, N 0, N 1, N 2 } size: {3} rootsizeleftrightleftrightleftright BinaryTreeN0N0 N1N1 N2N2 N1N1 N2N2 left right N0N0 : [N 0, 3, N 1, N 1, null, null, null, null]

15. 1/23/2003University of Virginia15 Search for each candidate vector, Korat: invokes repOk() and monitors the execution builds a field ordering (list of fields ordered by the accessed time) if repOk() returns true, output the structure if repOk() returns false, backtracks on the last accessed field, using the field ordering

16. 1/23/2003University of Virginia16 Search when repOk() returns false, the field ordering is: N1N1 N2N2 left right N0N0 |root, N 0.left, N 0.right| [N 0, 3, N 1, N 1, null, null, null, null] backtracking on N 0.right, gives the next candidate: (increments the field domain index for the field that is last in the field ordering) N1N1 N2N2 left right N0N0 [N 0, 3, N 1, N 2, null, null, null, null]

17. 1/23/2003University of Virginia17 Search N1N1 N2N2 left right N0N0 [N 0, 3, N 1, N 1, null, null, null, null] N1N1 N2N2 left right N0N0 [N 0, 3, N 1, N 2, null, null, null, null] with backtracking, Korat prunes 4 4 candidates of type: [N 0, 3, N 1, N 1, _, _, _, _]

18. 1/23/2003University of Virginia18 Nonisomorphism two candidates are isomorphic if:   ;  o, o’  O C,r ;  f  fields(o) ;  p  P. o.f == o’ in C  (o).f ==  (o’) in C’ and o.f == p in C  (o).f == p in C’ isomorphism => state space partitioned only the lexicographically smallest candidate is generated it is used to increment field domain indices by more than 1. N1N1 N2N2 left N0N0 N0N0 N2N2 N1N1

19. 1/23/2003University of Virginia19 Nonisomorphism-Algorithm N2N2 N1N1 left N0N0 N2N2 N1N1 right N0N0 |root, N 0.left, N 0.right, N 2.left, N 2.right| backtracking on a field f (pointer to object o f of class c f ): class domain: c f {N0, N1, N2} [N 0, 3, N 2, null, null, null, null, null] [N 0, 3, N 2, null, null, null, null, N 1 ] (?)

20. 1/23/2003University of Virginia20 Generating Test Cases to generate test inputs for method m, Korat builds a class that represents m’s inputs builds repOk() that checks m’s precondition generates all inputs that satisfy repOk() class BinaryTree_remove { //@ invariant repOk(); BinaryTree This; BinaryTree.Node n; boolean repOk() { return This.repOk() && This.has(n); } } class BinaryTree { //@ invariant repOk();... //@ requires has(n); void remove(Node n) {... } }

21. 1/23/2003University of Virginia21 Checking Correctness Korat uses: JML toolset for generating oracles JUnit for executing tests and reporting errors to test a method m, Korat invokes m on each input and test the output using the oracle

22. 1/23/2003University of Virginia22 Roadmap

23. 1/23/2003University of Virginia23 Results & Evaluation BenchmarkSizeState Space Structs Generated Time (sec) BinaryTree8 12 2 53 2 92 1430 208,012 1.53 233.59 HeapArray6868 2 20 2 29 13139 1,005,075 1.21 42.61 LinkedList8 12 2 91 2 150 4,140 4,213,597 1.32 690.00 TreeMap7979 2 92 2 130 35 122 8.81 2148.50 HashSet7 11 2 119 2 215 2386 277,387 3.71 926.71 AVTree52 50 598,35862.05

24. 1/23/2003University of Virginia24 Results & Evaluation BenchmarkMethodMax Size Test Cases Gen Time (sec) Test Time (sec) BinaryTreeremove3150.640.73 HeapArrayextractMax613,1390.871.39 LinkedListreverse280.670.76 TreeMapput819,912136.192.70 HashSetadd713,1063.901.72 AVTreelookup427,7344.3314.63

25. 1/23/2003University of Virginia25 Roadmap

26. 1/23/2003University of Virginia26 Questions & Comments non-Java environments? clear enough explanations for algorithms? proof for the search algorithm? paper quality: outstanding / good / bad / awful ? anything else you want to add?