1. Automatic Unit Tests Generation Tools SEPL Seminar Benny Pasternak December 2007

2. 2 Agenda Brief Introduction Automatic Generation Tools JCrasher Symstra GenUTest Summary

3. 3 Definition A unit is the smallest testable part of an application In the object oriented paradigm it is a class A unit test consists of a fixed sequence of method invocations with fixed arguments Unit test explores a particular aspect of the behavior of the Class Under Test (CUT)

4. 4 Yet Another Calculator Example public class Calculator { public int add(int a, int b) { return a + b; } public int subtract(int a, int b) { return a - b; } public int multiply(int a, int b) { return a * b; } public int divide(int a, int b) { return a / b; }

5. 5 Small JUnit Test public class CalculatorTest { @Test public void add3and5Equals8() { Calculator calc = new Calculator(); int result = calc.add(4, 6); assertEquals(result, 10); } @Test(expected= java.lang.ArithmeticException.class) public void divideByZeroThrowsException() { Calculator calc = new Calculator(); int result = calc.divide(3,0); } @Test public void subtract5from6Equals2() { Calculator calc = new Calculator(); int result = calc.subtract(6, 5); assertEquals(result, 2); }

6. 6 Test run and feedback

7. 7 Effective Unit Tests Effective unit test should be: Comprehensive – unit test should exercise all class methods and achieve high code coverage rate Test in Isolation – CUT dependent objects (e.g., Logger, Serializer ) should not be tested

8. 8 Comprehensiveness StackInt push(int) int pop() int top() bool empty() reverse() Unit Test test 1 (). test n () test 2 ()

9. 9 Isolation Logger Serializer StackInt push(int) int pop() int top() bool empty() reverse() LinkedList MockLinkedList MockLogger MockSerializer Unit Test test1() test2(). testn()

10. 10 Benefits Enables the immediate detection of bugs introduced into a unit whenever code changes occur  Unit tests provide a safety net of regression tests and validation tests which encourage developers to refactor existing code. Elimination of uncertainty in units  Simplifies integration Provides a sort of living documentation of the system

11. 11 However… Writing effective unit tests is a hard and tedious task… Lots of unit tests need to be written (unit tests code may be larger than project code) Provide tools to assist developers

12. 12 Tool Classification Frameworks – JUnit, NUnit Mock Frameworks – EasyMock, jMock Generation – Automatic generation of unit tests Selection – Selecting a small set of unit tests from a large one Prioritization – Deciding what the “best order” to run the tests

13. 13 Unit Test Generation Test Generation – generate a sequence of CUT method-calls Test Assertion – Provide an assertion to determine if the test result is correct

14. 14 Test Generation Techniques Random Execution random sequences of method calls with random values Symbolic Execution method sequences with symbolic arguments builds constraints on arguments produces actual values by solving constraints Capture & Replay capture real sequences seen in actual program runs or test runs

15. 15 Test Assertion Generation Techniques Manual (Design by Contract) Uncaught Exceptions Classifies a test as potentially faulty if it throws an unexpected exception Operation Model Infer an operational model from manual tests Properties: objects invariants, method pre/post conditions Properties violation  potentially faulty Capture & Replay Compare test results/state changes to the ones captured in actual program runs and classify deviations as possible errors.

16. 16 Generation Tool Map Random Execution Symbolic Execution Capture & Replay Uncaught Exceptions Operational Model Generation Selection Prioritization Eclat Symclat JCrasher Symstra Test Factoring SCRAPE Rostra CR Tool GenuTest Substra PathFinder Jartege JTest

17. 17 Tools Presented Today JCrasher (Random & Uncaught Exceptions) Symstra (Symbolic & User provided specifications) GenUTest (Capture & Replay)

18. 18 JCrasher – An Automatic Robustness Tester for Java (2003) Christoph Csallner Yannis Smaragdakis Available at http://www-tatic.cc.gatech.edu/grads/c/csallnch/jcrasher/

19. 19 Goal Robustness quality goal – “A class should not crash with an unexpected runtime exception, regardless of the parameters provided.” Do not assume anything about domain Robustness goal applies to all classes Function to determine class under test robustness: expected exception type  pass unexpected exception type  fail

20. 20 Parameter Space Huge parameter space. Example: m(int,int) has 2^64 param combinations Covering all parameters combination is impossible May not need all combinations to cover all control paths that throw an exception Pick a random sample Control flow analysis on byte code could derive parameter equivalence class

21. 21 Architecture Overview

22. 22 Type Inference Rules Search class under test for inference rules Transitively search referenced types Inference Rules Method Y.m(P1,P2,.., Pn) returns X: X  Y, P1, P2, …, Pn Sub-type Y {extend | implements } X: X  Y Constants: int  -1, 0,1 Add each discovered inference rule to mapping: X  inference rules returning X

23. 23 Generate Test Cases For a Method

24. 24 Exception Filtering JCrasher runtime catches all exceptions Example generated test case: Public void test1() throws Throwable { try { /* test case */ } catch (Exception e) { dispatchException(e); // JCrasher runtime } Uses heuristics to decide whether the exception is a Bug of the class  pass exception on to JUnit Expected exception  suppress exception

25. 25 Exception Filter Heuristics

26. 26 Symstra: Framework for Generating Unit Tests using Symbolic Execution (2005) Tao Xie Darko Wolfram David Marinov Schulte Notkin

27. 27 Binary Search Tree Example public class BST implements Set { Node root; int size; static class Node { int value; Node left; Node right; } public void insert (int value) { … } public void remove (int value) { … } public bool contains (int value) { … } public int size () { … } }

28. 28 Other Test Generation Approaches Straight forward – generate all possible sequences of calls to methods under test Cleary this approach generates too many and redundant sequences BST t1 = new Bst();BST t2 = new Bst(); t1.size();t2.size(); t2.size();

29. 29 Other Test Generation Approaches Concrete-state exploration approach Assume a given set of method calls arguments Explore new receiver-object states with method calls (BFS manner)

30. 30 Exploring Concrete States Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3) new BST() insert(1) insert(2) insert(3) remove(1) remove(2) remove(3) 1st Iteration 1 2 3

31. 31 Exploring Concrete States Method arguments: insert(1), insert(2), insert(3), remove(1), remove(2), remove(3) new BST() insert(1) insert(2) insert(3) remove(1) remove(2) remove(3) 2nd Iteration 1 2 1 2 3 1 3 insert(2)insert(3) remove(1) remove(2) remove(3)

32. 32 Generating Tests from Exploration Collect method sequences along the shortest path new BST() insert(1) insert(2) insert(3) remove(1) remove(2) remove(3) 2nd Iteration 1 2 1 2 3 1 3 insert(2)insert(3) remove(1) remove(2) remove(3) BST t = new BST(); t.insert(1); t.insert(3);

33. 33 Exploring Concrete States Issues Not solved state explosion problem Need at least N different insert arguments to reach a BST with size N experiments shows memory runs out when N = 7 Requires given set of relevant arguments in our case insert(1), insert(2), remove(1), …

34. 34 Concrete States  Symbolic States new BST() insert(1) insert(2) insert(3) remove(1) remove(2) remove(3) 1 2 1 2 3 1 3 insert(2)insert(3) remove(1) remove(2) remove(3) new BST() insert(x1) x1 x2 X1<x2 insert(x2)

35. 35 Symbolic Execution Execute a method on symbolic input values Inputs: insert(SymbolicInt x) Explore paths of the method Build a path condition for each path Conjunct conditionals or their negations Produce symbolic states ( ) For example x1 x2 X1<x2

36. 36 Exploring Symbolic States public void insert(SymbolicInt x) { if (root == null) { root = new Node(x); } else { Node t = root; while (true) { if (t.value < x) { // explore rigtht subtree } else if (t.value > x) { // explore left subtree } else return; } size++; } new BST() insert(x1) x1 x2 X1<x2 insert(x2) x1 x2 X1>x2 x1 X1= x2 S1 S2 S3 S4 S5

37. 37 Generating Tests from Exploration Collect method sequences along the shortest path Generate concrete arguments by using a constraint solver new BST() insert(x1) x1 x2 X1<x2 insert(x2) x1 x2 X1>x2 S1 S2 S3 S4 BST t = new BST(); t.insert(x1); t.insert(x2); BST t = new BST(); t.insert(-1000000); t.insert(-999999); X1>x2

38. 38 Results

39. 39 Results

40. 40 More Issues Symstra uses specifications (pre, post, invariants) written in JML. These are transformed to run-time assertions Limitations: can not precisely handle array indexes currently supports primitive arguments can generate non-primitive arguments as sequence of method calls. These eventually boil down to methods with primitive arguments

41. GenUTest: A Unit Test and Mock Aspect Generation Tool Benny Pasternak Shmuel Tyszberowicz Amiram Yehudai

42. 42 Motivation Writing effective unit tests is a hard and tedious process At maintenance phase, writing tests from scratch is not considered cost effective  Corollary: Maintenance remains a difficult process Goal: Automatically generate unit tests for projects in maintenance phase

43. 43 Example Developers are asked to create effective unit tests for an existing software project StackInt is an implementation of integers’ stack, with the operations: Push Pop Top Empty Reverse Goal is to test StackInt effectively

44. 44 Obtaining Test Cases From Existing Tests System/Module test that exercises IntStack as follows: Test can be used to obtain test cases for unit tests stack :IntStack new() push(2) push(3) reverse() addFirst(2) addFirst(3) lst :LinkedList 2 pop() removeFirst() 2 new() …

45. 45 GenUTest Captures and records execution of IntStack during module/system tests in order to obtain test cases Recorded events are used to generate unit tests for IntStack Unit tests assist developers in the testing process

46. 46 Example – Generated Unit Test 1 @Test public void testpop1() 2 { 3 // test execution statements 4 IntStack IntStack_2 = new IntStack(); // #1 5 IntStack_2.push(2); // #2 6 IntStack_2.push(3); // #3 7 IntStack_2.reverse(); // #4 8 int intRetVal6 = IntStack_2.pop();// #5 9 10 // test assertion statements 11 assertEquals(intRetVal6,2); 11 } Unit Test Code stack :IntStack new() push(2) push(3) reverse() addFirst(2) addFirst(3) lst :LinkedList 2 pop() removeFirst() 2 new() …

47. 47 Example Logger Serializer StackInt push(int) int pop() int top() bool empty() reverse() LinkedList Unit Test test1() test2(). testn()

48. 48 Aspect Oriented Programming class StackInt { void reverse() { LinkedList newlst = new LinkedList(); int size = lst.size(); for (int i = 0; i < size; i++) { int elem = lst.get(i); newlst.addFirst(elem); } lst = newlst; } int pop() { int elem = lst.removeFirst(); return elem; } Join points object instantiation method-calls field setter/getter Pointcut 1 Pointcut 2Around Advice print(“Before”); execute join point print(“After”); print(“Before”); print(“After”);

49. 49 AOP – Quick Summary Join points – well defined execution points in the control flow of the program (object instantiation, method-calls, field member access) Pointcut – expression that specifies a set of join points Advices – code specified to execute before, after, or around pointcuts Aspects – The equivalent to class. Holds pointcut declarations and advices

50. 50 Example Logger Serializer StackInt push(int) int pop() int top() bool empty() reverse() LinkedList Unit Test test1() test2(). testn() Pointcut 2 Pointcut 1 Mock Advice mock object behavior code Mock Advice mock object behavior code

51. Implementation 51

52. 52 Capture Phase Inter-object interactions logs Capture Code Program Code AspectJ Compiler Software is instrumented with capture functionality at constructor-calls, method- calls, field getter/setters returnVal = targetObject.someMethod(arg 1, … arg n ); signature target object signature arguments’ values return value, target object, arguments’ values return value/thrown exception Instrumented System – P’ Attributes of interactions captured: Inter-object interactions are captured and logged during runtime

53. 53 Capture Phase Instrumentation is performed using AspectJ More elegant and simpler mechanism However, it is a weaker mechanism than conventional instrumentation techniques that directly access a program’s Java bytecode  Requires the use of elegant workarounds to handle special cases:  non primitive arrays: obj1.peform(myArray[6]);  string syntactic: String me = “Benny”;

54. 54 Generation Phase – Step I Given a testable event, a backtracking algorithm recursively generates the statements needed for executing the test 1 @Test public void testpop1() { 2 // test execution statements 3 4 5 6 7 8 9 10 11 } IntStack IntStack_2 = new IntStack(); // #1 IntStack_2.push(2); // #2 IntStack_2.push(3); // #3 IntStack_2.reverse(); // #4 int intRetVal6 = IntStack_2.pop(); // #5

55. 55 Backtracking Algorithm Generally, in order to execute a test, GenUTest needs to generate statements that replay the relevant sequence of recorded events in a correct manner Execution of: intRetVal1 = obj1.process(obj2) Requires: obj1 and obj2 must be in the correct state

56. 56 Backtracking Algorithm Object states are represented by method-calls sequences: state T (o) = (method, method, … method ) Time is represented by a sequence number incremented before a method begins execution and after it finishes execution The interval [before, after] is called the method-interval t1t1 t2t2 tntn t 1< t 2 <..< t n <T

57. 57 Backtracking Algorithm Logged interactions: … obj1.report()[65,80] obj2.report()[51,64] obj1.process(obj2)[31,50] obj2.perform(obj3)[21,30] obj3.initialize()[9,20] obj2 = new Type2()[5,8] obj3 = new Type3()[3,4] obj1 = new Type1()[1,2] obj3obj2obj1Method Interval

58. 58 Backtracking Algorithm (cont) Generated statements: Algorithm may need to remove redundant statements Static and dynamic types of objects are stored for: casting – myObject = (MyObject)List.get(2); null values – obj1.process(null); static methods – System.out.println(“Hello World”); changes in modifier access policy – inner private class inheriting from a public outer one Type1 obj1 = new Type1(); Type3 obj3 = new Type3(); Type2 obj2 = new Type2(); obj3.initialize(); obj2.perform(obj3); int intRetVal1 = obj1.process(obj2);

59. 59 Generation Phase – Step II Case I – Value is returned from the call Generate statements that compare value test with value captured. Case II – An exception is thrown Generate statements that expect a particular exception 1 @Test public void testpop1() { 2 // test execution statements 3 4 5 6 7 8 9 // test assertion statements 10 11 } IntStack IntStack_2 = new IntStack(); // #1 IntStack_2.push(2); // #2 IntStack_2.push(3); // #3 IntStack_2.reverse(); // #4 int intRetVal6 = IntStack_2.pop(); // #5 assertEquals(intRetVal6,2);

60. 60 Mock Aspect Generation Definitions: Incoming method-calls – method-calls invoked by the unit test on the Class Under Test (CUT) Outgoing method-calls – method-calls invoked by the CUT on dependent objects Class Under Test Unit Test Object #1 ` Object #2 ` Object #n ` incoming method-calls outgoing method-calls

61. 61 Mock Aspect Generation Definitions: mi( A() ) – method interval of A() [Before A, After A ] method A() contains method B() if mi( A() ) contains mi( B() ) [Before A, After A ] [Before B, After B ] Observations: method B() resides in the control flow of method A() iff method A() contains method B() An outgoing method-call of the CUT is contained in exactly one incoming method-call ∩

62. 62 Mock Aspect Generation Last definition Outgoing( I() ) is the sequence I() is an incoming method call Io 1 (), Io 2 (), …, Io n () are all the outgoing method-calls contained in I() If method o() is contained in method I() and method o() is the j th element in Outgoing( I() ) then method o() is uniquely identified by the pair (mi( I() ), j)

63. 63 Mock Aspect Generation – Needed Example Four outgoing method-calls to addFirst() mi( push(2) ) is [5,8] Outgoing( push(2) ) = addFirst(2) is uniquely identified by mi( push(3) ) is [9,12] Outgoing( push(3) ) = addFirst(3) is uniquely identified by 12 13 IntStack_2 :IntStack new() push(2) push(3) reverse() addFirst(2) addFirst(3) get(0) lst :LinkedList newlst :LinkedList get(1) addFirst(2) 2 pop() removeFirst() 2 5 8 9 6 7 10 11 new() 1 4 2 3 addFirst(3) 3 2 Class Under Test Unit Test Object #1 ` Object #2 ` Object #n ` incoming method-calls outgoing method-calls 24

64. 64 Mock Aspect Generation – Needed Example Mi(reverse()) is [13,24] Outgoing(reverse()) = get(0) is uniquely identified by addFirst(3) is uniquely identified by get(1) is uniquely identified by addFirst(2) is uniquely identified by 12 13 IntStack_2 :IntStack new() push(2) push(3) reverse() addFirst(2) addFirst(3) get(0) lst :LinkedList newlst :LinkedList get(1) addFirst(2) 2 pop() removeFirst() 2 5 8 9 6 7 10 11 new() 1 4 2 3 addFirst(3) 3 2 Class Under Test Unit Test Object #1 ` Object #2 ` Object #n ` incoming method-calls outgoing method-calls 24

65. 65 Mock Aspect Generation Algorithm works as follows: 1. For each incoming method-call I() of the CUT, outgoing( I() ) is calculated 2. Each outgoing method-call is uniquely identified 3. For each incoming method-call I() different pointcut and advice are generated 4. A statement that sets method interval and clears the element counter is added before the incoming method call is invoked in the unit test 5. Bookkeeping code is added in advice 6. Backtracking algorithm is applied to mimic the behavior of the dependent object in the advice

66. 66 Mock Aspect Generation – Sample Code Integer around(): call (Object java.util.LinkedList.get(int)) && restriction() { if (before == 13 && after == 24) { if (elementCounter == 1) { elementCounter++; return 3; } if (elementCounter == 3) { elementCounter++; return 2; } thrown new RuntimeException(“Invalid method interval”); } void setMI(int b, int a) { before = b; after = a; elementCounter == 1; } @Test public void testpop1() { // test execution statements IntStack IntStack_2 = new IntStack(); IntStack_2.push(2); IntStack_2.push(3); StackIntMockAspect.setMI(13,24); IntStack_2.reverse(); int intRetVal6 = IntStack_2.pop(); // test assertion statements assertEquals(intRetVal6,2); } StackIntMockAspect.ajStackIntTest.java 4 3 5 5 6 6

67. 67 Implementation Overview Inter-object interactions logs Capture Code Program Code AspectJ Compiler Instrumented System – P’ Capture Phase Unit tests & Mock Aspects Generator Generation Phase Mock Aspects Unit Tests JUnit Test Results Unit Testing Phase AspectJ Compiler CUT weaved with mock advices

68. 68 Experimentation Used on open source project JODE (Java Optimize and Decompile Environment) http://jode.sourceforge.net/ http://jode.sourceforge.net/ JODE is a medium sized project ~35K loc Executed JODE combined with GenUTest on a chosen input GenUTest generated 592 unit tests from recorded data captured during runtime

69. 69 Experimentation Measured code coverage with EclEmma (www.eclemma.org/): 1. Execution of JODE on chosen input Coverage is 25% of JODE’s lines of code 2. Execution of generated unit tests with JUnit Coverage is 5.2% of JODE’s lines of code Current limitations and bugs may cause generation of invalid tests Primary reason for differences in loc coverage rate

70. 70 Limitations Partial support for inner classes and anonymous classes Does not support multi-thread applications Support of arrays need to be improved Scalability and performance issues

71. 71 Future Work Handle limitations and extend support: Inner/Anonymous classes, multi-threaded support, Optimize array handling, optimize performance Scalability – selective capturing, detect redundant tests, discard non mutating events, make use of concrete object states Research effectiveness in detecting regression bugs

72. 72 More tools not mentioned… JTest – commercial product by parasoft Jartege – operational model formed from JML PathFinder – symbolic execution tool by NASA Symclat – An evolution of Eclat & Symstra Rostra – Framework for detecting redundant object oriented unit tests Orstra – Augmenting Generated Unit-Test Suites with Regression Oracle Checking Agitator – www.agitar.com (Nice presentation available on website)www.agitar.com

73. 73 Summary Quick unit testing Survey Lots of unit testing tools Roughly categorized into frameworks, generation, selection and prioritization Concentrated on various test and assertion generation techniques used in several tools: JCrasher Symstra GenUTest

74. 74 Thank you for listening Questions?