1. Pex White Box Test Generation for.NET Nikolai Tillmann, Microsoft Research SMT 2008

2. 2 A unit test is a small program with assertions. void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3); Assert.IsTrue(set.Count == 2); } Many developers write such unit tests by hand. This involves determining a meaningful sequence of method calls, selecting exemplary argument values (the test inputs), stating assertions.

3. void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y); Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); } Parameterized Unit Tests separate two concerns: 1) The specification of externally visible behavior(assertions) 2) The selection of internally relevant test inputs(coverage) Parameterized Unit Testing bridges the gap between Unit Testing, and Design-By-Contract paradigm Parameterized Unit Testing bridges the gap between Unit Testing, and Design-By-Contract paradigm Parameterized Unit Testing are algebraic specifications!

4. Test input generator Pex starts from parameterized unit tests Generated tests are emitted as traditional unit tests Dynamic symbolic execution framework Symbolic execution based on monitoring and re-execution Whole-program, white-box code analysis At the level of the.NET instructions (bytecode) Support for “Java-like” programs as well as “unsafe” code SMT-solver Z3 determines satisfying assignments for constraint systems representing execution paths

7. How to test this code? (Real code from.NET base class libraries.) 7

8. 8

9. Main challenge: Making sure it does not crash by writing many tests that cover the code 9 Possible test case, written by Hand

10. Test input, generated by Pex 10

11. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve Initially, choose Arbitrary

12. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve Initially, choose Arbitrary a[0] = 0; a[1] = 0; a[2] = 0; a[3] = 0; … a[0] = 0; a[1] = 0; a[2] = 0; a[3] = 0; …

13. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve Initially, choose Arbitrary Path Condition: … ⋀ magicNum != 0x95673948 Path Condition: … ⋀ magicNum != 0x95673948

14. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve Initially, choose Arbitrary … ⋀ magicNum != 0x95673948 … ⋀ magicNum == 0x95673948 … ⋀ magicNum != 0x95673948 … ⋀ magicNum == 0x95673948

15. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve a[0] = 206; a[1] = 202; a[2] = 239; a[3] = 190; a[0] = 206; a[1] = 202; a[2] = 239; a[3] = 190; Initially, choose Arbitrary

16. Test Inputs Constraint System Execution Path Known Paths Run Test and Monitor Record Path Condition Choose an Uncovered Path Solve Initially, choose Arbitrary

17. Results in VS Report: Coverage, path conditions

18. 18 ldtokenPoint::GetX call__Monitor::EnterMethod brfalseL0 ldarg.0 call__Monitor::NextArgument<Point> L0:.try {.try { call__Monitor::LDARG_0 ldarg.0 call__Monitor::LDNULL ldnull call__Monitor::CEQ ceq call__Monitor::BRTRUE brtrueL1 call__Monitor::BranchFallthrough call__Monitor::LDARG_0 ldarg.0 … ldtokenPoint::X call__Monitor::LDFLD_REFERENCE ldfldPoint::X call__Monitor::AtDereferenceFallthrough brL2 L1: call__Monitor::AtBranchTarget call__Monitor::LDC_I4_M1 ldc.i4.m1 L2: call__Monitor::RET stloc.0 leaveL4 } catch NullReferenceException { ‘ call__Monitor::AtNullReferenceException rethrow } L4: leaveL5 } finally { call__Monitor::LeaveMethod endfinally } L5:ldloc.0 ret class Point { int x; int y; public static int GetX(Point p) { if (p != null) return p.X; else return -1; } } Prologue Epilogue Calls will perform symbolic computation Calls to build path condition Record concrete values to have all information when this method is called with no proper context (The real C# compiler output is actually more complicated.)

19. Similar to representation of verification conditions in ESC/Java, Spec#, … Terms for Primitive types (integers, floats, …) Constants Unary and binary expressions ‘struct’ types Tuples Instance fields of classes Mapping of references to values Elements of arrays, memory accesses through pointers Mapping of integers to values …

20. Goal: Efficient representation of evolving program states Reduction of ground terms to constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical operations Tries/Patricia Trees to represent associative-commutative-with- unit operators Normal form of polynomials Update trees Other simplification rules, e.g. \forall x. ceq(vtable(x, m1), m2) => ceq(objecttype(x), t) where m2 overrides m1, and t is the sealed declaring type of m2

21. Problem: Reachable code not known initially No loop invariants, loops must be unfolded Without guidance, symbolic execution may get stuck unfolding the same loop forever Solution: Search strategies outside of SMT solver choose “next branch to flip” Fair choice between different strategies Individual strategies based on program structure, including: Fair choice of branch instructions Fair choice of branch instructions + stack contexts Fair choice of branch coverage

22. Independent constraint optimization + Constraint caching (similar to EXE) Idea: Related execution paths give rise to "similar" constraint systems Example: Consider x>y ⋀ z>0 vs. x>y ⋀ z<=0 If we already have a cached solution for a "similar" constraint system, we can reuse it x=1, y=0, z=1 is solution for x>y ⋀ z>0 we can obtain a solution for x>y ⋀ z<=0 by reusing old solution of x>y: x=1, y=0 combining with solution of z<=0: z=0

23. Decision procedures for uninterpreted functions with equalities, linear integer arithmetic, bitvector arithmetic, arrays, tuples Support for universal quantifiers Used to model custom theories, e.g..NET type system Model generation Models used as test inputs Incremental solving Push / Pop of contexts for model minimization Programmatic API For small constraint systems, text through pipes would add huge overhead

24. Problem: Pex can collect constraints over private fields, constraint solver determines assignment for private fields How to bring object into desired state? Private fields cannot be initialized freely, but only through constructor and other methods Approach taken by Pex: Automatic selection of constructor and state-modifying methods based on static code analysis Exploration of constructor and methods to find non- exceptional paths

26. void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); } Assumptions and assertions are explored just like all other branches Executions which cause assumption violations are ignored, not reported as errors or test cases 26

27. AppendFormat(null, “{0} {1}!”, “Hello”, “World”);  “Hello World!”.Net Implementation: public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) { if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf = (ICustomFormatter)provider.GetFormat( typeof(ICustomFormatter)); … 27

28. Introduce a mock class which implements the interface. Write assertions over expected inputs, provide concrete outputs public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); } } Problems: Costly to write detailed behavior by example How many and which mock objects do we need to write? 28

29. Introduce a mock class which implements the interface. Let an oracle provide the behavior of the mock methods. public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult (); return o; } } Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 29

31. We applied Pex on a core.NET component Already extensively tested for several years Assertions written by developers >10,000 public methods >100,000 basic blocks Sandbox Restriction of access to external resources (files, registry, unsafe code, …) 10 machines (P4, 2Ghz, 2GB RAM) running for 3 days Exploration started from simple, generated parameterized unit tests (one per public method); assertions embedded in code 31

32. Coverage achieved: 43% block coverage 36% arc coverage Errors found: A significant number of benign errors, e.g. NullReferenceException, IndexOutOfRangeException, … 17 unique errors involving violation of developer-written assertions, exhaustion of memory, other serious issues. 32

33. 33 ClassnameBlocksHitArcsHit A (mostly stateless methods)>30095%>40090% B (mostly stateless methods)>10097%>20094% C (stateful)>20076%>30065% D (parsing code)>50081%>80073% E (numerical algorihm)>40071%>60067% F (numerical algorihm)>10082%>20079% G (numerical algorihm)>10098%>10097% H (numerical algorihm)>20071%>20061% I (numerical algorihm) >20097%>30096% Automatically achieved coverage on selected classes for core.NET component

34. Assumption: Environment is deterministic "Environment" includes all code that is not monitored, e.g. native code, uninstrumented code Pex prunes non-deterministic behavior Assumption: Program is single-threaded Potential solution: control and explore thread scheduling like all other test inputs Limitations of constraint solver Z3 has no built-in theories for floating point arithmetic approximation with rationals (linear arithmetic only) Bounds on Z3's time and memory consumption 34

35. Goal: Test-input generation for programs with contracts (preconditions, postconditions, invariants, etc.) In Verisoft project, compiler generates Boogie or MSIL programs from C code annotated with contracts MSIL programs embed most contracts in executable form These contracts are turned into constraints by Pex, which performs a path-sensitive analysis Challenge: Non-executable contracts Quantifiers: may range over “all integers” or “all pointers” Predicates for memory-safety: do not translate directly into machine-observable behavior

36. Better scalability More sophisticated search-frontiers (e.g. based on fitness function that determines distance to target state) Summarizing execution paths instead of exploring them (TACAS'08) Inference of likely invariants/contracts (DySy, ICSE'08) Dealing with multi-threaded programs Controlling the scheduler Systematically exploring all relevant thread interleavings Race detection Tom Ball et. al. are building such analyses on Pex framework (ManagedChess)

37. Program model checkers JPF, Kiasan/KUnit (Java), XRT (.NET) Combining random testing and constraint solving DART (C), CUTE (C), EXE (C), jCUTE (Java), SAGE (X86) … 37

38. 38 Parameterized Unit Tests separate two concerns Specification of externally visible behavior Selection of test inputs to cover internal behavior Pex automates test input generation Uses SMT-solver Z3 Dynamic Symbolic Execution platform for.NET Used internally in Microsoft to test core.NET components Pex is publicly available for academic use. http://research.microsoft.com/Pex http://research.microsoft.com/Pex

40. Dynamic symbolic execution will systematically explore the conditions in the code which the constraint solver understands. And happily ignore everything else, e.g. Calls to native code Difficult constraints (e.g. precise semantics of floating point arithmetic) Result: Under-approximation, which is appropriate for testing Calls to external world Unmanaged x86 code Unsafe managed.NET code (with pointers) Safe managed.NET code Most interesting programs are beyond the scope of static symbolic execution.

41. When generating test inputs for any method, e.g. DateTime ParseDateTime(string s) { … } a regression test suite can be generated, where each test asserts the observed behavior. void ParseDateTimeTest132() { DateTime result = ParseDateTime(“6/19/2008”); Assert(result.ToString() == “06/19/2008”); }

42. XRT: Exploring Runtime Interpreter for.NET programs Static symbolic execution Used Simplify to determine unsatisfiability of path constraints Successful for self-contained programs Used today on a large scale within Microsoft for quality assurance purposes as the core of the model-based testing tool “Spec Explorer 2007”. Does not work well for real-world programs All environment behavior must be modeled Modeling of entire environment is often not feasible