1. A Test Case + Mock Class Generator for Coding Against Interfaces Mainul Islam, Christoph Csallner Software Engineering Research Center (SERC) Computer Science and Engineering Department University of Texas at Arlington, USA February 07, 2011. Published at: 8th International Workshop on Dynamic Analysis (WODA), 2010

2. Test case generation Dynamic Symbolic Execution Mock Class Motivating examples to generate Mock Class Algorithm to generate Mock Class Results Demonstration Mock Class Generator for Coding Against Interfaces 1 Talk Outline

3. What is the goal behind testing ? Mock Class Generator for Coding Against Interfaces 2 Test Case Generation public void Test(int x) { if (x > 0) { if (x == 1234567890) { throw new Exception(“Bug”); } What can be a good test case to test this method ? Probability to reach the bug (with a random test input) is: ½ * 1/2^31

4. Explore all feasible execution paths Initialize the input with symbolic values and execute the program over symbolic values At conditional statements check if either of the branches can be taken (using a constraint solver) For each path we get an accumulated path condition Mock Class Generator for Coding Against Interfaces 3 Symbolic Execution If (s) then … else … C falsetrue C` = C ⋀ sC` = C ⋀⌝ s s Dynamic symbolic execution: Combine Concrete and Symbolic execution

5. void TestMe (int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”); } This example is taken from the slides of Nikolai Tillman (with permission) 4 Dynamic Symbolic Execution by Example a == null a.length > 0 a[0] == 123 T T T F F F Constraints to SolveInput Observed Constraints null {} {0} {123} a != null && !(a.length > 0) a == null a != null && a.length > 0 && a[0] != 123 a != null && a.length > 0 && a[0] == 123 a != null a != null && a.length > 0 a != null && a.length > 0 && a[0] == 123 SolveExecute Choose Next Path a == null Done: No path left

6. public interface A { // … } public class C { public void foo (A a, int x) { // … } Mock Class Generator for Coding Against Interfaces 5 Introduction to Mock Class

7. When we might need to generate mock-classes ? At the initial stage of any development Mock Class Generator for Coding Against Interfaces 6

8. Test case generation is important Interfaces are very important Existing techniques are not very good at this Pex - can generate an instance of a class that implements an interface that is used in the code * * http://research.microsoft.com/en-us/projects/pex/http://research.microsoft.com/en-us/projects/pex/ Mock Class Generator for Coding Against Interfaces 7 Why is this important ?

9. Goal: Support interfaces in test case generation for object-oriented programs Mock Class Generator for Coding Against Interfaces 8

10. Motivating Example 1 public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } Mock Class Generator for Coding Against Interfaces 9 To reach this block of code, ‘a’ must be an instance of A, as well as an instance of B public interface A { int m1(int x); } public interface B { int m2(int x); }

11. Motivating Example 2 public class K implements A, B { // … } public class C { public void bar (A a, int x) { if (a == null) return; if ( !(a instanceof B) ) { B b = (B) a; int z = b.m2(x); } Mock Class Generator for Coding Against Interfaces 10 K implements all of the interfaces referred by the bar method, but fails reach the code block public interface A { int m1 (int x); } public interface B { int m2 (int x); }

12. Framework DSC – Dynamic Symbolic Execution Engine (for Java) Z3 – Constraint Solver http://research.microsoft.com/en-us/um/redmod/projects/z3/ Mock Class Generator for Coding Against Interfaces 11

13. Our approach to generate Mock Class Determining the type of the Mock Class Generate meaningful method body if needed Mock Class Generator for Coding Against Interfaces 12

14. Sub-/Supertype relation in Java Java defines a binary subtype relation If type B implements/extends type A then, - A is a direct super type of B - B is a direct sub type of A Reflexive: A is also a subtype of itself Transitive: if B is a subtype of A and C is a subtype of B then C is also a subtype of A Mock Class Generator for Coding Against Interfaces 13

15. Sub-/Supertype relation in Java (Cont.) A class has one direct class super type and arbitrarily many interface super types. Exceptions: type object – has no direct super type type null – has no direct sub type Mock Class Generator for Coding Against Interfaces 14

16. Subtype constraints Mock Class Generator for Coding Against Interfaces 15 Object ABC null Object ABC M null Initial types in the system A desired solution: with new type M public interface A { int m1 (int x); } public interface B { int m2(int x); } public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } type(a) subtypeof A type(a) != null type type(a) subtypeof B Constraints:

17. Subtype Relation Matrix public interface A { int m1 (int x); } public interface B { int m2(int x); } public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } Mock Class Generator for Coding Against Interfaces 16 NullObjectABCM 0Nullxxxxxx 1Objectx 2Axx 3Bxx 4Cxx 5MxMAMBMCx NullObject 0Nullxx 1Objectx NullObjectABC 0Nullxxxxx 1Objectx 2Axx 3Bxx 4Cxx NullObjectABCM 0Nullxxxxxx 1Objectx 2Axx 3Bxx 4Cxx 5Mxxxx Solution: MA = true MB = true MC = false

18. Program execution Found Solution? No Yes Algorithm: Generating Mock Class Dynamic Symbolic Data Structure Repair 17 Build Subtype Relation matrix (Generate the constraints) NO_MC < MAX_MC Generate Test Case Introduce a new Mock Class, N0_MC++ Encode the Mock Class in the type system No Yes Solution not found Yes Initialization: NO_MC = 0 MAX_MC = 2

19. Generating Method Body: for Example 1 Mock Class Generator for Coding Against Interfaces 18 public class M implements A, B { int m1 (int x){ return 0; } int m2(int x){ return 0; } Outline of the source code for Mock Class M public interface A { int m1 (int x); } public interface B { int m2(int x); } public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); if (z >= 10) { //… } return 10;

20. Results We have run our solution on JMS (Java messaging system) These are all hand-written test cases Mock Class Generator for Coding Against Interfaces 19 Interfaces used from JMS Number of goals Number of goals reached w/o mock- classes Number of goals reached with mock- classes Message 606 ByteMessage 505 MapMessage 404 StreamMessage 404 TextMessage 505 ObjectMessage 404

21. Comparison with Pex: Motivating Example 1 public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); } Mock Class Generator for Coding Against Interfaces 20 public interface A { int m1(int x); } public interface B { int m2(int x); } Goal 1 Goal 2 PEX reaches 1/2 goal(s) DSC reaches 2/2 goal(s)

22. Future Work Extend the solution to support more types, e.g. ‘Array’ Determine a upper bound to the number of mock classes needed to solve a system Evaluation Mock Class Generator for Coding Against Interfaces 21

23. Future Work (Cont.) Mock Class Generator for Coding Against Interfaces 22 public class M implements A, B { int m1 (int x){ return 0; } int m2(int x){ return 0; } Outline of the source code for Mock Class M public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) { B b = (B) a; int z = b.m2(x); if (z >= 10) { //… } return 10; Extend method body generation for complex types

24. Mock Class Generator for Coding Against Interfaces 23 Demo

25. Mock Class Generator for Coding Against Interfaces 24 Thank you !