Tutorial Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research) Tao Xie (North Carolina State University)

 Unit Testing  Parameterized Unit Testing  Input Generation with Pex  Patterns for Parameterized Unit Testing  Parameterized Models  Wrapping up

 We will use the tool Pex as part of the exercises  To install it, you need  Windows XP or Vista .NET 2.0  Download Pex from  Two variations :  pex.devlabs.msi : requires Visual Studio 2008 Team System  pex.academic.msi : does not require Visual Studio, but for non-commercial use only  To run the samples, you need VS Team System

Introduction

void ReadWrite() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); } void ReadWrite() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count); }  A unit test is a small program with assertions  Test a single (small) unit of code

 Design and specification By Example  Documentation  Short feedback loop  Code coverage and regression testing  Extensive tool support for test execution and management  Happy path only  Hidden integration tests  Touch the database, web services, requires multiple machines, etc…  New Code with Old Tests  Redundant tests

 Coverage: Are all parts of the program exercised?  statements  basic blocks  explicit/implicit branches  …  Assertions: Does the program do the right thing?  test oracle Experience:  Just high coverage of large number of assertions is no good quality indicator.  Only both together are!

Goal: Implement and test TrimEnd:  Open Visual Studio  Create CodeUnderTestProject, and implement TrimEnd  File -> New -> Project -> Visual C# -> Class Library  Add new StringExtensions class  Implement TrimEnd  Create TestProject, and test TrimEnd  File -> New -> Project -> C# -> Test -> Test Project  Delete ManualTest.mht, UnitTest1.cs, …  Add new StringExtensionsTest class  Implement TrimEndTest  Execute test  Test -> Windows -> Test View, execute test, setup and inspect Code Coverage // trims occurences of the ‘suffix’ from ‘target’ string TrimEnd(string target, string suffix); // trims occurences of the ‘suffix’ from ‘target’ string TrimEnd(string target, string suffix);

public class StringExtensions { public static string TrimEnd(string target, string suffix) { if (target.EndsWith(suffix)) target = target.Substring(0, target.Length-suffix.Length); return target; } public class StringExtensions { public static string TrimEnd(string target, string suffix) { if (target.EndsWith(suffix)) target = target.Substring(0, target.Length-suffix.Length); return target; } [TestClass] public class StringExtensionTest { [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = StringExtensions.TrimEnd(target, suffix); Assert.IsTrue(result == "Hello "); } [TestClass] public class StringExtensionTest { [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = StringExtensions.TrimEnd(target, suffix); Assert.IsTrue(result == "Hello "); }

Introduction

var list = new List(); list.Add(item); var count = list.Count; var list = new List(); list.Add(item); var count = list.Count; Assert.AreEqual(1, count); } Assert.AreEqual(1, count); }  Three essential ingredients:  Data  Method Sequence  Assertions void Add() { int item = 3; void Add() { int item = 3;

 Which value matters?  Bad choices cause redundant, or worse, incomplete test suites.  Hard-coded values get stale when product changes.  Why pick a value if it doesn’t matter? list.Add(3);

void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); } void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count); }  Parameterized Unit Test = Unit Test with Parameters  Separation of concerns  Data is generated by a tool  Developer can focus on functional specification

 A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); } void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData); }  string name, string data: name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name,WriteResource(name,data)), data)  string name, string data: name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(name,WriteResource(name,data)), data)

Parameterized Unit Tests (PUTs) commonly supported by various test frameworks .NET: Supported by.NET test frameworks    …  Java: Supported by JUnit 4.X  Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex   Java: Supported by Agitar AgitarOne 

Goal: Create a parameterized unit test for TrimEnd..  Refactor: Extract Method [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = ParameterizedTest(target, suffix); Assert.IsTrue(result == "Hello "); } static string ParameterizedTest(string target, string suffix) { var result = StringExtensions.TrimEnd(target, suffix); return result; } [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = ParameterizedTest(target, suffix); Assert.IsTrue(result == "Hello "); } static string ParameterizedTest(string target, string suffix) { var result = StringExtensions.TrimEnd(target, suffix); return result; }

17 Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible Observations:  Reachability not decidable, but approximations often good enough  Encoding of functional correctness checks as assertions that reach an error statement on failure

void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); }  Assumptions and assertions induce branches  Executions which cause assumption violations are ignored, not reported as errors or test cases

 Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Semi - Random: QuickCheck  Cheap, Fast  “It passed a thousand tests” feeling  Dynamic Symbolic Execution: Pex, Cute,EXE  Automated white-box  Not random – Constraint Solving

Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!= a!=null && a.Length>0 && a[0]== Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T T F T F F Execute&Monitor Solve Choose next path Done: There is no path left. Negated condition

 Open CoverMe project in Visual Studio.  Right-click on CoverMe method, and select “Run Pex”.  Inspect results in table. public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } public static void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); }

 How to test this code? (Actual code from.NET base class libraries) 22

[PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } [PexClass, TestClass] [PexAllowedException(typeof(ArgumentNullException))] [PexAllowedException(typeof(ArgumentException))] [PexAllowedException(typeof(FormatException))] [PexAllowedException(typeof(BadImageFormatException))] [PexAllowedException(typeof(IOException))] [PexAllowedException(typeof(NotSupportedException))] public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } } }

 If the test does not throw an exception, it succeeds.  If the test throws an exception,  (assumption violations are filtered out),  assertion violations are failures,  for all other exception, it depends on further annotations.  Annotations  Short form of common try-catch-assert test code  [PexAllowedException(typeof(T))]  [PexExpectedException(typeof(T))]

A brief overview

Z3 Pex ExtendedReflection  Instrumentation framework “Extended Reflection”  Instruments.NET code at runtime  Precise data flow and control flow analysis Code Under Test

Z3 Pex ExtendedReflection  Pex listens to monitoring callbacks  Symbolic execution along concrete execution Code Under Test

Z3 Pex ExtendedReflection  Constraint solving with SMT solver Z3  Computes concrete test inputs Code Under Test

Z3 Pex ExtendedReflection  Execute with test inputs from constraint solver  Emit test as source code if it increases coverage Code Under Test

Z3 Pex ExtendedReflection  ExtendedReflection: Code Instrumentation  Z3: SMT constraint solver  Pex: Dynamic Symbolic Execution Code Under Test

ldtokenPoint::GetX call__Monitor::EnterMethod brfalseL0 ldarg.0 call__Monitor::Argument<Point> L0:.try {.try { call__Monitor::LDARG_0 ldarg.0 call__Monitor::LDNULL ldnull call__Monitor::CEQ ceq call__Monitor::BRTRUE brtrueL1 call__Monitor::BranchFallthrough call__Monitor::LDARG_0 ldarg.0 … ldtokenPoint::X call__Monitor::LDFLD_REFERENCE ldfldPoint::X call__Monitor::AtDereferenceFallthrough brL2 L1: call__Monitor::AtBranchTarget call__Monitor::LDC_I4_M1 ldc.i4.m1 L2: call__Monitor::RET stloc.0 leaveL4 } catch NullReferenceException { ‘ call__Monitor::AtNullReferenceException rethrow } L4: leaveL5 } finally { call__Monitor::LeaveMethod endfinally } L5:ldloc.0 ret class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } } class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } } Prologue Epilogue Calls will perform symbolic computation Calls to build path condition Record concrete values to have all information when this method is called with no proper context (The real C# compiler output is actually more complicated.) p p null p == null

Goal: Show how to use Extended Reflection callbacks to build a tracer applications  Pex ships with a samples project called Tracer.  Tracer listens to EnterMethod, ExitMethod events  Log arguments values

Pex’s representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, … Terms for  Primitive types (integers, floats, …), constants, expressions  Struct types by tuples  Instance fields of classes by mutable ”mapping of references to values"  Elements of arrays, memory accessed through unsafe pointers by mutable “mapping of integers to values" Efficiency by  Many reduction rules, including reduction of ground terms to constants  Sharing of syntactically equal sub-terms  BDDs over if-then-else terms to represent logical operations  Patricia Trees to represent AC1 operators (including parallel array updates)

 SMT-Solver (“Satisfiability Modulo Theories”)  Decides logical first order formulas with respect to theories  SAT solver for Boolean structure  Decision procedures for relevant theories: uninterpreted functions with equalities, linear integer arithmetic, bitvector arithmetic, arrays, tuples  Model generation for satisfiable formulas  Models used as test inputs  Limitations  No decision procedure for floating point arithmetic 

Parameterized Unit Testing

Test Project Code-Under-Test Project Code-Under-Test Project Parameterized Unit Tests Pex Generated Tests

 Right-click on the code-under-test project  In the Solution Explorer  Select “Pex”  Select “Create Parameterized Unit Test Stubs”

Generated Test Project References Microsoft.Pex.Framework (and more) References Microsoft.Pex.Framework (and more) References VisualStudio UnitTestFramework References VisualStudio UnitTestFramework Contains source files with generated Parameterized Unit Test stubs

 Right-click on Parameterized Unit Test  For now, it just calls the implementation, but the you should edit it, e.g. to add assertions  Select “Run Pex Explorations”

Exploration Status Exploration Status Current Parameterized Unit Test Issue bar Important messages here !!! Issue bar Important messages here !!! Input/Output table Row = generated test Column = parameterized test input or output Input/Output table Row = generated test Column = parameterized test input or output

Test outcome filtering Test outcome filtering Passing Test Passing Test Fix available New Test Failing Test Failing Test

Exception Stack trace Exception Stack trace Review test Allow exception

Pex Explorer makes it easy to digest the results of many Parameterized Unit Tests, and many generated test inputs

pex.exe

 Rich information, used by Pex developers  Full event log history  Parameter table  Code coverage  etc… HTML rendered from XML report file: It is easy to programmatically extract information! HTML rendered from XML report file: It is easy to programmatically extract information!

 Tell Pex which Type you are testing  Code coverage  Exception filtering  Search prioritization [PexClass(typeof(Foo))] public class FooTest { … [PexClass(typeof(Foo))] public class FooTest { …

 3 Domains  UserCodeUnderTest  UserOrTestCode  SystemCode public void CodeUnderTest() { … } [PexClass] public class FooTest { [PexMethod] public void ParamTest(…) { … [PexClass] public class FooTest { [PexMethod] public void ParamTest(…) { … public class String { public bool Contains(string value) { … public class String { public bool Contains(string value) { …

 Namespace, type, method filters  pex.exe test.dll /tf:Foo /mf:Bar  Partial match, case insensitive  Full match: add “!”: /tf:Foo!  Prefix: add “*”: /tf:Foo*  Many other filters  e.g. to select a “suite”: /sf=checkin  For more information: pex.exe help filtering

Goal: Explore input generation techniques with TrimEnd  Write or discuss a random input generator for TrimEnd  Adapt your parameterized unit tests for Pex  Start over with “Create Parameterized Unit Test Stubs”, or… ▪ Add reference to Microsoft.Pex.Framework ▪ Add using Microsoft.Pex.Framework; ▪ Add [PexClass(typeof(StringExtensions))] on StringExtensionsTest ▪ Add [PexMethod] on your parameterized tests  Right-click in test and ‘Run Pex Explorations’  Compare test suites

It’s called Parameterized Unit Testing

 The yellow event bar notifies about important events, including certain limitations Click on issue kind for more information

 You should act on these events:  Refactor your code, or  tell Pex to ignore it in the future,  let Pex analyze (“instrument”) more code, if possible. Click on a particular event for more information Understand the issue, and take an action

 If Pex reports that some code was uninstrumented, you may tell Pex to instrument and analyze it (if possible)

 Code instrumentation on Demand  Instrumentation has high performance overhead  Some parts of the code better ignored  Use PexInstrument… attributes  Pex will often suggest and insert those attributes for you [assembly: PexInstrumentAssembly(“Foo”)]

 Pex understand managed.NET code only  Pex does not understand native code.  Problem if branching over values obtained from the environment  Pex may not automatically detect all such cases. if (!File.Exists(f)) throw... File System?

 Pex analyzes every executed.NET instruction  Some used libraries may be surprisingly expensive to analyze  XML parsing  repeatedly converting data between different representations void Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } void Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); } Don’t do this.

There are decision procedures for individual path conditions, but…  Number of potential paths grows exponentially with number of branches  Reachable code not known initially  Without guidance, same loop might be unfolded forever

 To deal with the path explosion problem, Pex uses search heuristics.  Heuristics try to create “diversity”, i.e. they prefer paths which are so far underrepresented e.g. by their coverage vectors

 In the presence of loops and recursion, Pex could search forever, so Pex has many ways to bound the search  Named arguments of [PexMethod]  Pex will often suggest and update those bounds for you. [PexMethod(MaxConstraintSolverTime = 4)]

 Configurable bounds include:  TimeOut  MaxBranches  MaxCalls  MaxConditions ▪ Number of conditions that depend on test inputs  MaxRuns  ConstraintSolverTimeOut  ConstraintSolverMemoryLimit

 Division and modulo are expensive to reason about  No decision procedures for floating- point arithmetic  Pex currently uses heuristic search techniques for floating-point constraints

Test Inputs Constraint System Execution Path Known Paths Initially, choose Arbitrary Path-constraint solving is just hard. Reachability is undecidable! (Loops) (With external functions and dynamic loading, summaries are difficult.)

 Pex only explores single-threaded code  Related approach to explore thread- schedules (but not input parameters): CHESS

 Write assertions and Pex will try to break them  Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.  Assertions leveraged in product and test code  Pex can leverage Code Contracts (discussed later)

 Pex only uses public methods to configure non-public object fields  Heuristics built-in to deal with common types  User can help if needed void (Foo foo) { if (foo.Value == 123) throw … void (Foo foo) { if (foo.Value == 123) throw … [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar); } [PexFactoryMethod] Foo Create(Bar bar) { return new Foo(bar); }

Goal: Understand limitations of DSE.  PexGoal / PexExpectedGoals  Apply Pex to if (File.Exists(fileName)) PexGoal.Reached(); if (File.Exists(fileName)) PexGoal.Reached(); if (DateTime.Parse(s).Year == 2009) PexGoal.Reached(); if (DateTime.Parse(s).Year == 2009) PexGoal.Reached(); class Foo { private Foo() {} } … if (foo != null) PexGoal.Reached(); class Foo { private Foo() {} } … if (foo != null) PexGoal.Reached();

Tips and tricks

Problem: Constraint solver determines test inputs = initial state of test Most classes hide their state (private fields) State is initialized by constructor, and can be mutated only by calling methods What sequence of method calls reaches a given target state? There may be no such sequence In general, undecidable Two approaches: (Guided) exploration of constructor/mutator methods Testing with class invariants

Specification: [PexMethod] public void ArrayListTest(ArrayList al, object o) { PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o); }

 Factory methods  Parameterized factories create and configure objects  Explicit: Public static method annotated with [PexFactoryMethod]  Implicit: Pex guesses “best” constructor/setters  Result: Exploration of reachable states  Reachable using factory method  Reachable within configured bounds  Under-approximation of possible states

We will now describe how you can create objects via the exploration of a class invariant.  Write class invariant as boolean-valued parameterless method  Refers to private fields  Must be placed in implementation code  Write special constructor for testing only  May be marked as "debug only"  Constructor sets fields, assumes invariant  Result: Exploration of feasible states  May include states that are not reachable

Open project “ArrayListSample”.  Explore AddTestExplicit  Explore AddTestImplicit  Inspect generated test cases  Notice constructor implicitly used by Pex

 Edit code of ArrayList class:  Add “ad-hoc” Invariant method:  Add DEBUG-only constructor that allows to configure object freely, and then checks invariant: private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; } private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; } #if DEBUG public ArrayList(object[] items, int size) { this._items = items; this._size = size; if (!this.Invariant()) throw new Exception(); } #endif #if DEBUG public ArrayList(object[] items, int size) { this._items = items; this._size = size; if (!this.Invariant()) throw new Exception(); } #endif

Tips and tricks

 Assume, Arrange, Act, Assert [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert } [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert }

 For an API f(x), f -1 (f(x)) = x for all x void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped); } void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped); }

 For an API f(x), f -1 (f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed); } void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed); }

 For an API f(x), f -1 (f(f -1 (x)) = f -1 (x) for all x void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped); } void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped); }

 Observe a state change void ContainedAfterAdd(string value) { var list = new List (); list.Add(value); Assert.IsTrue(list.Contains(value)); } void ContainedAfterAdd(string value) { var list = new List (); list.Add(value); Assert.IsTrue(list.Contains(value)); }

 Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x. public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); } public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); }

 Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold that f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x. string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); } string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); }

 Allowed exception -> negative test case [PexAllowedException(typeof(ArgumentException))] void Test(object item) { var foo = new Foo(item) // validates item [PexAllowedException(typeof(ArgumentException))] void Test(object item) { var foo = new Foo(item) // validates item // generated test (C#) [ExpectedException(typeof(ArgumentException))] void Test01() { Test(null); // argument check } // generated test (C#) [ExpectedException(typeof(ArgumentException))] void Test01() { Test(null); // argument check }

 Indicate which portions of a PUT should be reachable. [PexExpectedGoals] public void InvariantAfterParsing(string input) { ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant(); } [PexExpectedGoals] public void InvariantAfterParsing(string input) { ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant(); }

 Generated test asserts any observed value  Return value, out parameters, PexGoal  When code evolves, breaking changes in observable will be discovered int AddTest(int a, int b) { return a + b; } int AddTest(int a, int b) { return a + b; } void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); } void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); }

Goal: Identify and apply patterns  Apply patterns to parameterized unit tests of TrimEnd.

 Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.  Integration test: exercises large portions of a system.  Observation: Integration tests are often “sold” as unit tests  White-box test generation does not scale well to integration test scenarios.  Possible solution: Introduce abstraction layers, and mock components not under test

AppendFormat(null, “{0} {1}!”, “Hello”, “World”);  “Hello World!”.Net Implementation: public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) { if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf = (ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …

 Introduce a mock class which implements the interface.  Write assertions over expected inputs, provide concrete outputs public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); } }  Problems:  Costly to write detailed behavior by example  How many and which mock objects do we need to write?

 Introduce a mock class which implements the interface.  Let an oracle provide the behavior of the mock methods. public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult (); return o; } }  Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 92

 Show AppendFormat code  Show AppendFormat test  Run Pex 93

 Chosen values can be shaped by assumptions public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult (); PexAssume.IsTrue(o is ICustomFormatter); return o; } }  (Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.) 94

 Choices to build parameterized models class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }} class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}

Goal: Test a ‘CopyFile’ method with a File System model  Write a CopyFile method using IFileSystem  Implement IFileSystem with System.IO.File and use it to test Copy.  Write a parameterized model for IFileSystem and use it to test Copy. interface IFileSystem { string ReadFile(string fileName); void WriteFile(string name, string content); }

Design By Contracts meets Automated Whitebox Testing

  Library to state preconditions, postconditions, invariants  Supported by two tools:  Static Checker  Rewriter: turns Code Contracts into runtime checks  Pex analyses the runtime checks  Contracts act as Test Oracle  Pex may find counter examples for contracts  Missing Contracts may be suggested

If present, Pex leverages [ContractInvariantMethod] to create objects via reflection and checking if invariant holds:  Install Code Contracts: Add reference to Microsoft.Contracts library  Add empty [ContractInvariantMethod]  Run Pex, click on “Add Invariant”  Repeat, edit manually [ContractInvariantMethod] protected void Invariant() { Contract.Invariant(this._items != (object[])null); Contract.Invariant( (uint)(this._size) <= (uint)(this._items.Length)); } [ContractInvariantMethod] protected void Invariant() { Contract.Invariant(this._items != (object[])null); Contract.Invariant( (uint)(this._size) <= (uint)(this._items.Length)); }

 Code Contracts allow specification of interface contracts  Pex‘ stubs framework can leverage contracts to check input and restrict output of mock objects.  Consider the following interfaces annotated with contracts (using Spec# notation): interface IFormatProvider { object GetFormat(Type formatType) requires formatType != null; ensures result != null && formatType.IsAssignableFrom(result.GetType()); }

 Create IFormatProvider interface with Code Contracts in code-under-test project  Create (parameterized) test project  Show stubs  Create parameterized unit test using IFormatProvider  Explore parameterized unit test

 Random input generators: many  Combining random testing and constraint solving  DART, CUTE, EXE, KLEE, CREST (C)  jCUTE, jFuzz (Java)  SAGE (X86)  …  Program model checkers  JPF, Kiasan/KUnit (Java), XRT (.NET)  Commercial tools  AgitarOne, … 103