1. Parameterized Unit Testing ICTSS’11 Tutorial Nikolai Tillmann Microsoft Research

2. Outline  First part: 14:00 - 15:00pm  Second part: 15:30 - 17:00pm

3. Outline  Unit Testing  Isolation  Parameterized Unit Testing  Data Generation by Dynamic Symbolic Execution  Patterns  Limitations and other Details

4. About the exercises…  Interactive: http://pexforfun.comhttp://pexforfun.com  Demos: Pex for Visual Studio http://research.microsoft.com/pex http://research.microsoft.com/pex  Requires Windows,.NET 2.0 or 4.0, ideally Visual Studio 2008 / 2010

5. Unit Testing

6. Quiz: Unit testing  What is a unit test?

7. Unit Testing void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); } void AddAndCount() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); }  A unit test is a small program with assertions  Test a single (small) unit of code  Let’s play a game!

8. The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; } string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; } t:\myapp.ini A=B Foo=C C=D t:\myapp.ini A=B Foo=C C=D

9. Quiz: Coverage  How much block coverage do we need? 1. 50% 2. 80% 3. 100% 4. Block coverage alone is not enough

10. Quiz: Coverage  How much block coverage do we need? 1. 50% 2. 80% 3. 100% 4. Block coverage alone is not enough

11. Quiz: Coverage  Do we need more tests to get 100% cov.? [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue(); } [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[0]); Reader.ReadFooValue(); } [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue(); } [TestMethod] void ExistingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“Foo=b”}); Reader.ReadFooValue(); }

12. Quiz: Assertions  Why write Assertions (or not)? 1. Documentation 2. Double check your program 3. Please your manager 4. Prevent future bugs 5. Validate user inputs 6. Catch errors early

13. Quiz: Assertions  Why write Assertions (or not)? 1. Documentation 2. Double check your program 3. Please your manager 4. Prevent future bugs 5. Validate user inputs 6. Catch errors early

14. Quiz: Assertions  Which Assertions should you write? 1. Assert.IsTrue(value == “b”); 2. Assert.IsTrue(value == null); 3. Assert.IsTrue(String.IsNullOrEmpty(value)) 4. Assert.IsTrue(false); 5. No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); } [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); }

15. Quiz: Assertions  Which Assertions should you write? 1. Assert.IsTrue(value == “b”); 2. Assert.IsTrue(value == null); 3. Assert.IsTrue(String.IsNullOrEmpty(value)) 4. Assert.IsTrue(false); 5. No assertions [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); } [TestMethod] void MissingFoo() { File.WriteAllLines(@"t:\myapp.ini",new string[]{“a=b”}); string value = Reader.ReadFooValue(); Assert.IsTrue(????); }

16. Quiz: Coverage + Assertions  What gives you confidence in the code? 1. High coverage, no assertions 2. Low coverage, many assertions 3. High coverage, many assertions 4. Low coverage, no assertions 5. I wrote it

17. Quiz: Coverage + Assertions  What gives you confidence in the code? 1. High coverage, few assertions 2. Low coverage, many assertions 3. High coverage, many assertions 4. Low coverage, no assertions 5. I wrote it

18. The Code Under Test string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; } string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; }

19. Quiz: Isolation  In the example, what are the external dependencies? 1. Network Share 2. Local Disk 3. No file system, all in memory string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini");... } string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini");... }

20. Quiz: Isolation  In the example, what are the external dependencies? 1. Network Share 2. Local Disk 3. No file system, all in memory string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini");... } string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini");... }

21. Quiz: Isolation  What is the problem with a Local Disk? 1. Mapping already exists 2. Cannot run tests concurrently 3. Disk full 4. Access rights

22. Quiz: Isolation  What is the problem with a Local Disk? 1. Mapping already exists 2. Cannot run tests concurrently 3. Disk full 4. Access rights

23. Quiz: Isolation  How do you mitigate the Local Disk issues? 1. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration 2. Refactoring: use Streams 3. Refactoring: introduce IFileSystem 4. Refactoring: pass the lines as parameter 5. Change implementation of File.ReadAllLines

24. Quiz: Isolation  How do you mitigate the Local Disk issues? 1. Always run on the same machine, same hardware, same credentials, same time, same temperature, same solar system configuration 2. Refactoring: use Streams 3. Refactoring: introduce IFileSystem 4. Refactoring: pass the lines as parameter 5. Change implementation of File.ReadAllLines

25. Quiz: Definition of Unit Test  What is a Unit Test?  A Unit Test is a program that runs fast the code under test, without environment dependencies, with assertions.  What is a Unit Test Suite?  A set of Unit Tests which achieves high code coverage

26. Isolation

27.  How do you change File.ReadAllLines ? 1. Override static method 2. Changing the.NET Framework source code (and recompiling it) 3. Rewrite application in JavaScript 4. Code instrumentation Dependency hell var lines = File.ReadAllLines(@"t:\v.ini");

28. Replace any.NET method with a delegate var lines = File.ReadAllLines(@"t:\myapp.ini"); File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn) { return new string[0]; }; File.ReadAllLines = delegate(string fn) MFile.ReadAllLinesString = delegate(string fn) { return new string[0]; }; What if we could replace File.ReadAllLines ? Moles

29. Motivation for Moles  Why another isolation framework?  Specifically designed to enable Pex  Simple, well-defined semantics ▪ “Replace any.NET method”

30. Mole Types Code Generation // System.IO public static class File { public static string[] ReadAllLines(string fn); } // System.IO public static class File { public static string[] ReadAllLines(string fn); } // System.IO.Moles public class MFile { public static Func ReadAllLinesString; } // delegate R Func (T t); // System.IO.Moles public class MFile { public static Func ReadAllLinesString; } // delegate R Func (T t);

31. Injecting Detours at Runtime // System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code } // System.IO public static class File { public static string[] ReadAllLines(string fn) { if (MFile.ReadAllLinesString != null) return MFile.ReadAllLines(fn); … original code } } Automatically injected at runtime Automatically injected at runtime

32. Moles Demo  Start from ReadFooValue problem  Create test project  Add moles for mscorlib  Write test using moles  Run test  Debug test  You can pick up project from ReadFooValueIsolation\ReadFooValueIsolation.sln

33. Quiz: Moles Usage  When should you use Moles (and not)? 1. Always use Moles to solve isolation issues 2. With Moles, one does not need to use interfaces anymore 3. Moles only should be used for 3 rd party API, use interfaces for isolation in your APIs 4. Moles can be used in production code 5. Moles lets you get away with untestable APIs 6. Moles make test cases more robust 7. With Moles, you do not need integration tests anymore 8. Moles make tests easier to understand 9. Moles is for poor programmers, real programmers rely on interfaces

34. Quiz: Moles Usage  When should you use Moles (and not)? 1. Always use Moles to solve isolation issues 2. With Moles, one does not need to use interfaces anymore 3. Moles only should be used for 3 rd party API, use interfaces for isolation in your APIs 4. Moles can be used in production code 5. Moles lets you get away with untestable APIs 6. Moles make test cases more robust 7. With Moles, you do not need integration tests anymore 8. Moles make tests easier to understand 9. Moles is for poor programmers, real programmers rely on interfaces

35. Exercise (Optional / homework)  Step-by-step tutorial:  research.microsoft.com/documentation.aspx research.microsoft.com/documentation.aspx  “Unit Testing with Microsoft Moles”  “Unit Testing SharePoint Foundation with Microsoft Pex and Moles”

36. What you learned so far  The Definition of Unit Testing  Unit Test Isolation 1. Identify External Dependencies 2. Replace each External API with your delegate

37. Parameterized Unit Testing

38. The Recipe of Unit Testing var list = new List(); list.Add(item); var count = list.Count; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); } Assert.AreEqual(1, count); }  Three essential ingredients:  Data  Method Sequence  Assertions void AddAndCount() { int item = 3; void AddAndCount() { int item = 3;

39. Quiz: list.Add(???)  Which value matters? 1. 0 2. 1 3. int.MaxValue, int.MinValue 4. -1 5. 1000 6. it does not matter 7. I don’t know until I read the code list.Add(???);

40.  Parameterized Unit Test = Unit Test with Parameters  Separation of concerns  Specification of behavior  Data to achieve coverage Parameterized Unit Testing void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void AddAndCount(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } for any list, for any item, for any list, for any item, … adding 1 item increments Count by 1

41. Parameterized Unit Tests are Algebraic Specifications  A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); } void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); } forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data) forall. string name, string data: name not_equal null and data not_equal null implies equals( ReadResource(name,WriteResource(name,data)), data)

42. Parameterized Unit Testing is going mainstream Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by.NET test frameworks  http://www.mbunit.com/  http://www.nunit.org/  …  Java: Supported by JUnit 4.X  http://www.junit.org/ Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex  http://research.microsoft.com/Pex/  Java: Supported by Agitar AgitarOne  http://www.agitar.com/

43. Problem We cannot execute parameterized unit tests without data. Where does the data come from?  Random data generator  Real customer data  Ranges  Some values hand-picked by dev/tester

44. Data Generation by Dynamic Symbolic Execution

45. 45 Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible How would you do it? Data Generation Challenge

46. Combines ideas from  symbolic execution  to capture behavior symbolically  model checking  to perform systematic bounded state space exploration  testing  to monitor and check concrete behavior  constraint solving  to find inputs Dynamic Symbolic Execution Combines many ideas

47. 47 Combines concrete and symbolic execution Algorithm: Set J := ∅ (J is set of already analyzed program inputs) Loop Choose program input i ∉ J (stop if no such i can be found) Output i Execute P(i); record path condition C (in particular, C(i) holds) Set J := J ∪ C (viewing C as the set { i | C(i ) } ) End loop Loop does not terminate if number of execution paths is infinite (in the presence of loops/recursion) Dynamic Symbolic Execution The high-level algorithm This choice decides search order Search order decides how quick we can achieve high code coverage! Incomplete constraint-solver leads to under-approximation This choice decides search order Search order decides how quick we can achieve high code coverage! Incomplete constraint-solver leads to under-approximation

48. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Dynamic Symbolic Execution Example

49. http://pex4fun.com/CoverMe

51. Dynamic Symbolic Execution Exercises  ArrayIndexLength Pex knows about all implicit, exception- throwing control-flow branches  ArrayHeap Pex models the heap  Assert, Assert123 Assertions connect code coverage and correctness http://pex4fun.com/DynamicSymbolicExecutionExercises

52. Assertions vs. Coverage  What about finding faults? void AssertIsTrue(bool condition) { if (!condition) throw new AssertionException(); }  Assertions induce branches  Pex tries to cover branches ▪ Pex tries to fail assertions -> finds bug!

53.  Create new project in Visual Studio.  Insert CoverMe method below.  Right-click on CoverMe method,  and select “Run Pex”.  Inspect results in table.  Save tests, …, have fun! public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Dynamic Symbolic Execution with Pex in Visual Studio

55. Generated Test Inputs are persisted as C# Unit Tests Generated Test Inputs are persisted as C# Unit Tests

56. Exercise (Optional / homework)  Step-by-step tutorial:  research.microsoft.com/documentation.aspx research.microsoft.com/documentation.aspx  “Exploring Code with Microsoft Pex”

57.  How to test this code? (Actual code from.NET base class libraries) Motivation: Unit Testing Hell ResourceReader 57

58. Motivation: Unit Testing Hell ResourceReader

59. Demo ResourceReader [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } }

60.  Defined by execution environment / programming language, symbolic execution precision, and constraint solving  Execution environment: C, Java, x86,.NET,…  Precision: linear vs. non-linear arithmetic, “gods integers” vs. bitvectors, concrete heap vs. symbolic heap., floating-point values, etc.  Solvers: lp_solve, CVCLite, STP, Disolver, Z3,…  Examples of DSE implementations:  DART (Bell Labs), and also CUTE “concolic execution”  EXE/EGT/KLEE (Stanford) “constraint-based execution”  Vigilante (Microsoft) to generate worm filters  BitScope (CMU/Berkeley) for malware analysis  Sage (Microsoft) for security testing of X86 code  Yogi (Microsoft) to verify device drivers (integrated in SLAM)  Pex (Microsoft) for parameterized unit testing of.NET code  CREST, jCUTE, jFuzz, … Dynamic Symbolic Execution Many implementations

61. Constraint Solving vs. Search Strategies Test Inputs Constraint System Execution Path Known Paths Initially, choose Arbitrary Path-constraint solving is just hard. Reachability is undecidable! (Loops)

62. Representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, … Terms for  Primitive types (integers, floats, …), constants, expressions  Struct types by tuples  Instance fields of classes by mutable ”mapping of references to values"  Elements of arrays, memory accessed through unsafe pointers by mutable “mapping of integers to values" Efficiency by  Many reduction rules, including reduction of ground terms to constants  Sharing of syntactically equal sub-terms  BDDs over if-then-else terms to represent logical operations  Patricia Trees to represent AC1 operators (including parallel array updates) Symbolic State Representation

63.  SMT-Solver (“Satisfiability Modulo Theories”)  Decides logical first order formulas with respect to theories  SAT solver for Boolean structure  Decision procedures for relevant theories: uninterpreted functions with equalities, linear integer arithmetic, bitvector arithmetic, arrays, tuples  Model generation for satisfiable formulas  Models used as test inputs  Limitations  No decision procedure for floating point arithmetic  Pex uses Z3: http://research.microsoft.com/z3http://research.microsoft.com/z3 Constraint Solving

64. Arithmetic Array Theory Uninterpreted Functions Satisfiability Modulo Theories void Foo(int x, int y, int[] a) { if (x + 2 == y) { a[x] = 3; Debug.Assert(f(a[y - 2]) == f(y – x + 1)); } void Foo(int x, int y, int[] a) { if (x + 2 == y) { a[x] = 3; Debug.Assert(f(a[y - 2]) == f(y – x + 1)); }

65. Pex uses some heuristic approaches for domain-specific problems.  Strings [TACAS’09, ICST’10]  “uninterpreted functions” for string operations  two-phase solving: first solve integer constraints involving indices and lengths, then character constraints  Floating-point arithmetic  Search-based approach: minimization of fitness function  … (more to come) Domain-specific Constraint Solving

66. Writing Parameterized Unit Tests

67. string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; } string ReadFooValue() { string[] lines = File.ReadAllLines(@"t:\myapp.ini"); foreach (var line in lines) { int index = line.IndexOf('='); string name = line.Substring(index); if (name == "Foo") { string value = line.Substring(index + 1); return value; } return null; } Exercise Write PUT for ReadFooValue Can you come up with Parameterized Unit Tests?

68.  Create new solution with C# class library  Write code for ReadFooValue  Create Test Project  Add moles for mscorlib  Write parameterized unit tests.  Or get result from ReadFooValuePUT\ReadFooValuePUT.sln Exercise Write PUT for ReadFooValue [PexMethod] // attribute marks pex tests public void Test(...) {... } [PexMethod] // attribute marks pex tests public void Test(...) {... }

69. Exercise Crash Test [PexMethod] void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); } [PexMethod] void Crash(string[] lines) { MFile.ReadAllLines = () => lines; Reader.ReadFooValue(); }  Execute the code without assertions

70. Exercise Assert Observed Results [PexMethod] public string Regression(string[] lines) { MFile.ReadAllLines = () => lines; return Reader.ReadFooValue(); } [PexMethod] public string Regression(string[] lines) { MFile.ReadAllLines = () => lines; return Reader.ReadFooValue(); }  Manually review outputs  Pex inserts Assertions automatically

71. Exercise [PexMethod] void FooExist(string value1) { PexAssume.IsTrue(value1 != null); var lines = new string[]{“Foo=“ + value1}; MFile.ReadAllLines = () => lines; var value2 = Reader.ReadFooValue(); PexAssert.AreEqual(value1, value2); } [PexMethod] void FooExist(string value1) { PexAssume.IsTrue(value1 != null); var lines = new string[]{“Foo=“ + value1}; MFile.ReadAllLines = () => lines; var value2 = Reader.ReadFooValue(); PexAssert.AreEqual(value1, value2); }  Write and read property

72. Test Project Test Generation Work-Flow Code-Under-Test Project Code-Under-Test Project Parameterized Unit Tests Pex Generated Tests

73. Patterns

74. Patterns how to create objects

75. [PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); } What are the test inputs?  Only null  An array list created by calling a constructor ... and also calling Add(…) many times  … and also calling Clear() many times …… Quiz: Complex Objects As Inputs

76. [PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); } Other possible problems: What if…  ArrayList has no public constructor  ArrayList is an abstract class (or interface) Quiz: Complex Objects As Inputs

77. Problem:  Test parameters may be classes  Symbolic execution tracks constraints over (private) fields of objects  Constraint solver determines solution, assigning concrete values to (private) fields  However, in practice objects must be initialized by constructor, and can be mutated only by calling certain methods  What sequence of method calls reaches a desired field assignment?  There may be no such sequence  In general, undecidable Creating complex objects The Problem

78.  Factory methods  Parameterized factories create and configure objects  Explicit: Public static method annotated with [PexFactoryMethod]  Implicit: Pex guesses “best” constructor/setters  Result: Exploration of reachable states  Reachable using factory method  Reachable within configured bounds  Under-approximation of possible states Object creation: Factory methods

79. Hand-written factory method: [PexFactoryMethod(ArrayList)] public static ArrayList Create( int capacity, object[] values) { var a = new ArrayList(capacity); foreach (value in values) a.Add(value); return a; } Pex will explore both factory method and PUT! Factory methods ArrayList

80. Open project “ArrayListSample”.  Explore AddTestExplicit  Explore AddTestImplicit  Inspect generated test cases  Notice constructor implicitly used by Pex  Save factory method  Edit factory method  Explore again Demo: Factory methods

81.  Write class invariant as boolean-valued parameterless method  Refers to private fields  Must be placed in implementation code  Write special constructor/factory method for testing only  Sets fields, and assumes invariant  Result: Exploration of feasible states  May include states that are not reachable Alternative Object creation via class invariants

82. Patterns for Parameterized Unit Tests

83. Pattern 4A  Assume, Arrange, Act, Assert [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act // assert quiz 1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); } [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act // assert quiz 1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }

84. Pattern 4A  Assume, Arrange, Act, Assert [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act // assert quiz 1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); } [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act // assert quiz 1. Assert(target != null); 2. Assert(target.Count == count + 1); 3. Assert(target[0] == value); }

85. Pattern Regression Tests  Generated test asserts any observed value  Return value, out parameters  When code evolves, breaking changes in observable will be discovered int AddTest(int a, int b) { return a + b; } int AddTest(int a, int b) { return a + b; } void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); } void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); }

86. Pattern Roundtrip  For an API f(x), f -1 (f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); // assert quiz 1. Assert.AreEqual(value, s); 2. Assert.AreEqual(s, parsed); 3. Assert.AreEqual(parsed, value); } void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); // assert quiz 1. Assert.AreEqual(value, s); 2. Assert.AreEqual(s, parsed); 3. Assert.AreEqual(parsed, value); }

87. Pattern Roundtrip  For an API f(x), f -1 (f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); // assert quiz 1. Assert.AreEqual(value, s); 2. Assert.AreEqual(s, parsed); 3. Assert.AreEqual(parsed, value); } void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); // assert quiz 1. Assert.AreEqual(value, s); 2. Assert.AreEqual(s, parsed); 3. Assert.AreEqual(parsed, value); }

88. Pattern Normalized Roundtrip  For an API f(x), f -1 (f(f -1 (x)) = f -1 (x) for all x void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); // assert quiz 1. Assert(x == intermediate); 2. Assert(intermediate == roundtripped); 3. Assert(normalized == roundtripped); 4. Assert(x == roundtripped); } void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); // assert quiz 1. Assert(x == intermediate); 2. Assert(intermediate == roundtripped); 3. Assert(normalized == roundtripped); 4. Assert(x == roundtripped); }

89. Pattern Normalized Roundtrip  For an API f(x), f -1 (f(f -1 (x)) = f -1 (x) for all x void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); // assert quiz 1. Assert(x == intermediate); 2. Assert(intermediate == roundtripped); 3. Assert(normalized == roundtripped); 4. Assert(x == roundtripped); } void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); // assert quiz 1. Assert(x == intermediate); 2. Assert(intermediate == roundtripped); 3. Assert(normalized == roundtripped); 4. Assert(x == roundtripped); }

90. Pattern Reachability  Indicate which portions of a PUT should be reachable. [PexAssertReachEventually] public void Constructor(object input) { new Foo(input); PexAssert.ReachEventually(); } [PexAssertReachEventually] public void Constructor(object input) { new Foo(input); PexAssert.ReachEventually(); }

91. Pattern Same Observable Behavior  Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x. public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); } public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); }

92. Pattern Commutative Diagram  Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold that f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x. string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); } string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); }

93. Homework Patterns  Read patterns paper:  research.microsoft.com/Pex/documentation.aspx research.microsoft.com/Pex/documentation.aspx  “Parameterized Test Patterns for Microsoft Pex”

94. Limitations It’s called Parameterized Unit Testing

95.  Pex understand managed.NET code only  Pex does not understand native code.  Problem if branching over values obtained from the environment  Pex may not automatically detect all such cases. Testability if (!File.Exists(f)) throw... File System?

96. Hidden Complexity  Pex analyzes every executed.NET instruction  Some used libraries may be surprisingly expensive to analyze  XML parsing  repeatedly converting data between different representations void Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } void Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } Don’t do this.

97. Exploration Boundaries  Configurable bounds include:  TimeOut  MaxBranches  MaxCalls  MaxConditions ▪ Number of conditions that depend on test inputs  MaxRuns  ConstraintSolverTimeOut  ConstraintSolverMemoryLimit

98. Multi-threaded code  Unlike test inputs, thread-interleavings can normally not be controlled  Thus, Pex can only explore single-threaded code  Related approach to explore thread- schedules (but not input parameters) by controlling thread-scheduler: CHESS http://research.microsoft.com/CHESS http://research.microsoft.com/CHESS

99. Lack of Test Oracle  Unit Testing != Test Generation

100. Wrapping up

101. Conclusion What you learned today  The Definition of Unit Testing  Unit Tests are not Integration Tests  Unit Test Isolation  Introduction of Abstraction Layers  Parameterized Unit Testing  Separation of concerns: Coverage/Specification  Patterns to write Parameterized Unit Tests  Data Generation by Dynamic Symbolic Execution

102. Availability  Pex and Moles are Visual Studio 2010 Power Tools (academic/commercial license available)  Minimum: Windows XP,.NET 2.0  Ideally: Visual Studio 2010 Professional  http://research.microsoft.com/pex/ http://research.microsoft.com/pex/  http://pexforfun.com http://pexforfun.com

103. Thank you http://research.microsoft.com/pex http://sites.google.com/site/asergrp/