1. How unit testing frameworks hurt your unit testing efforts
(and what you can do about it)
Dror Helper
Consultant & software architect
CodeValue

2. About.Me Consultant & Architect @CodeValue
Developing software since 2002
Mocking code since 2008
Test driven developer & clean coder
Pluralsight author
Blogger:

3. Let’s play: “Guess the unit testing framework” game

4. [Test] public void AddItem_AddTwoValidProductsToCart_TotalEqualsItemsPrice() { var fakeRepository = A.Fake<IProductRepository>(); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); A.CallTo(() => fakeRepository.GetProductById(1)).Returns(product1)); var shoppingCart = new ShoppingCart(fakeRepository); shoppingCart.AddItem(1); shoppingCart.AddItem(2); Assert.That(shoppingCart.Total, Is.EqualTo(300)); }

5. @Test public void addProductById_AddTwoProducts_GetTotalPriceReturnSumPrice() { IProductRepository fakeRepository = mock(IProductRepository.class); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); when(fakeRepository.getProductById(1)).thenReturn(product1); when(fakeRepository.getProductById(1)).thenReturn(product2); ShoppingCart cart = new ShoppingCart(fakeRepository); cart.AddItem(1); cart.AddItem(2); assertEquals(300.0, cart.getTotal(), 0); }

6. TEST(ShoppingCartTests, AddProductById_AddTwoProducts_GetTotalPriceReturnSumPrice) { FakeRepository fake_repository; Product product1 = Product(1, "product-1", 100); Product product2 = Product(2, "product-12, 200); EXPECT_CALL(fake_repository, GetProductById(1)).WillRepeatedly(Return(product1)); EXPECT_CALL(fake_repository, GetProductById(2)).WillRepeatedly(Return(product2)); ShoppingCart shopping_cart(fake_repository); shopping_cart.AddProductById(1); shopping_cart.AddProductById(2); ASSERT_EQ(300, shopping_cart.GetTotalPrice()); }

7. Which was the first unit testing framework?
NUnit (.NET)
SUnit (Smalltalk)
JUnit (Java)
CppUnit (C++)
2002
1994*
1997
2000 X √

8. xUnit testing frameworks history
1991
1994
1997
2000
2002
Taligent
framework
KentBeck
writes first
version of
SUnit (
SmallTalk
) test
framework*
Kent Beck &
Erich
Gamma
Create
JUnit
During a flight
to OOPSLA
Feathers ports
Michael
JUnit to C++ (
CppUnit )
NUnit
2.0

9. Why you need a unit testing framework
Declare Tests
Organize tests in Fixtures
Verify results using Assertions
Execute tests using Runner(s)

10. [Test] public void AddItem_AddTwoValidProductsToCart_TotalEqualsItemsPrice() { var fakeRepository = A.Fake<IProductRepository>(); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); A.CallTo(() => fakeRepository.GetProductById(1)).Returns(product1)); var shoppingCart = new ShoppingCart(fakeRepository); shoppingCart.AddItem(1); shoppingCart.AddItem(2); Assert.That(shoppingCart.Total, Is.EqualTo(300)); }

11. Year 1997 in programming C++98 is one year away Java 1 C#/.NET
Major additions in the release on February 19, 1997 included:[5]
an extensive retooling of the AWT event model
inner classes added to the language
JavaBeans
JDBC
RMI
reflection which supported Introspection only, no modification at runtime was possible.
JIT (Just In Time) compiler on Microsoft Windows platforms, produced for JavaSoft by Symantec
Internationalization and Unicode support originating from Taligent[6]

12. C++17 Java 8 C# 7 WHAT HAPPENED SINCE 1997
Major additions in the release on February 19, 1997 included:[5]
an extensive retooling of the AWT event model
inner classes added to the language
JavaBeans
JDBC
RMI
reflection which supported Introspection only, no modification at runtime was possible.
JIT (Just In Time) compiler on Microsoft Windows platforms, produced for JavaSoft by Symantec
Internationalization and Unicode support originating from Taligent[6]

13. How unit testing frameworks changed since 1997?

14. Unit Tests should be: How does your Unit Testing framework helps you?
Quick to write
Simple to read
Trivial to maintain
How does your Unit Testing framework helps you?

15. [Test] public void AddItem_AddTwoValidProductsToCart_TotalEqualsItemsPrice() { var fakeRepository = A.Fake<IProductRepository>(); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); A.CallTo(() => fakeRepository.GetProductById(1)).Returns(product1)); var shoppingCart = new ShoppingCart(fakeRepository); shoppingCart.AddItem(1); shoppingCart.AddItem(2); Assert.That(shoppingCart.Total, Is.EqualTo(300)); }
Arrange

16. [TestMethod] public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { var user1 = new User {ImageUrl = " Reputation = 10}; var user2 = new User {ImageUrl = " Reputation = 10}; var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); var viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository)); await viewModel.LoadUser(); var result = await InvokeAsync(() => ((SolidColorBrush)viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result); }

17. private UserDetailsViewModel _viewModel; [TestInitialize] public async Task InitilizeUserViewModel() { var user1 = new User { ImageUrl = " Reputation = 10 }; var user2 = new User { ImageUrl = " Reputation = 10 }; var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); _viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository)); } [TestMethod] public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { await _viewModel.LoadUser(); var result = InvokeAsync(() => ((SolidColorBrush)_viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);

18. private UserDetailsViewModel _viewModel; [TestInitialize] public async Task InitilizeUserViewModel() { var user1 = new User { ImageUrl = " Reputation = 10 }; var user2 = new User { ImageUrl = " Reputation = 10 }; var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); _viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository)); } [TestMethod] public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { await _viewModel.LoadUser(); var result = InvokeAsync(() => ((SolidColorBrush)_viewModel.ReputationTrend).Color); Assert.AreEqual(Colors.White, result);

19. (ab)using fields in tests
[Test] public async Task GetTextsWhenServerDownloadFail() { // Who am I??? var result = await _surveyService.GetTextsAsync(); Assert.IsFalse(result.HasError); Assert.NotNull(result.TheResult); }

20. [TestMethod] public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { var user1 = CreateUser(reputation: 10); var user2 = CreateUser(reputation: 10); var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); var viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository)); await viewModel.LoadUser(); var result = await InvokeAsync(() => viewModel.ReputationTrend.Color); Assert.AreEqual(Colors.White, result); }

21. Over abstraction [TestMethod]
public async Task LoadUser_ReputationStaysTheSame() {
var viewModel = InitializeSystem(10, 10);
await viewModel.LoadUser();
CheckColor(Colors.White, viewModel); }

22. The problem with factory methods
private User CreateUser(int reputation) {
return new User {
ImageUrl = "
Reputation = reputation }; }
private User CreateUser(int reputation, string imageUrl) {
return new User {
ImageUrl = imageUrl,
Reputation = reputation }; }

23. [TestMethod] public async Task LoadUser_ReputationStaysTheSame_ReputationTrendSame() { var user1 = CreateUser(reputation: 10); var user2 = CreateUser(reputation: 10); var fakeUserRepository = new FakeUserRepository(new[] { user1, user2 }); var viewModel = await InvokeAsync(() => new UserDetailsViewModel(fakeUserRepository)); await viewModel.LoadUser(); var result = await InvokeAsync(() => viewModel.ReputationTrend.Color); Assert.AreEqual(Colors.White, result); }

24. Solution: Builder pattern
class UserBuilder { private int _id, _reputation; private string _displayName, _imageUrl; public UserBuilder() { _id = 1; _displayName = "dummy"; _imageUrl = " } User Build() { return new User { Id = _id, DisplayName = _displayName, ImageUrl = _imageUrl, Reputation = _reputation };
Solution: Builder pattern

25. Solution: Builder pattern
class UserBuilder { ... public UserBuilder WithName(string displayName) { _displayName = displayName; return this; } public UserBuilder WithReputation(int reputation) { _reputation = reputation;
Solution: Builder pattern
var user1 = new UserBuilder()
.WithReputation(10)
.Build();

26. Using automocking container
Test
Container
SUT
New()
Configure
Create SUT
Create
Act

27. Automocking with AutoFixture
[Fact] public void YellIfTouchHotIron() { var fixture = new Fixture().Customize(new AutoFakeItEasyCustomization()); Fake<IMouth> fakeMouth = fixture.Freeze<Fake<IMouth>>(); Fake<IHand> fakeHand = fixture.Freeze<Fake<IHand>>(); A.CallTo(() => fakeHand.FakedObject.TouchIron(A<Iron>._)).Throws<BurnException>(); var brain = fixture.Create<Brain>(); brain.TouchIron(new Iron {IsHot = true}); A.CallTo(() => fakeMouth.FakedObject.Yell()).MustHaveHappened(); }

28. [Test] public void AddItem_AddTwoValidProductsToCart_TotalEqualsItemsPrice() { var fakeRepository = A.Fake<IProductRepository>(); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); A.CallTo(() => fakeRepository.GetProductById(1)).Returns(product1)); var shoppingCart = new ShoppingCart(fakeRepository); shoppingCart.AddItem(1); shoppingCart.AddItem(2); Assert.That(shoppingCart.Total, Is.EqualTo(300)); }
Act

29. How to decide what to test?
[Test] public void MyTest() { }

30. Solution: best practices and experience
Find out what is a “unit of work” means – in your project
Stick to known best practices
Learn and improve

31. [Test] public void AddItem_AddTwoValidProductsToCart_TotalEqualsItemsPrice() { var fakeRepository = A.Fake<IProductRepository>(); Product product1 = new Product(1, "product-1", 100); Product product2 = new Product(2, "product-12, 200); A.CallTo(() => fakeRepository.GetProductById(1)).Returns(product1)); var shoppingCart = new ShoppingCart(fakeRepository); shoppingCart.AddItem(1); shoppingCart.AddItem(2); Assert.That(shoppingCart.Total, Is.EqualTo(300)); }
Assert

32. Using different assertions
Assert.AreEqual(5, 2 + 2);
Assert.IsTrue(2 + 2, 5)

33. Using different assertions #2
Assert.AreEqual(5, 2 + 2);
Assert.IsTrue(2 + 2 == 5)

34. hidden assertions
StringAssert.Contains( string expected, string actual ); StringAssert.StartsWith( string expected, string actual ); CollectionAssert.AreEquivalent CollectionAssert.Contains Assert.That(result, ???);

35. [Test] public void CheckCompare() { var myClass = new MyClass(); Expect.That(() => myClass.ReturnFive() == 10); } public void CollectionContains() var c1 = new[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 }; Expect.That(() => c1.Contains(41));
Assert Helper

36. Can you spot the problem?
[TestMethod] public void PerformSomeActionReturns42() { var myClass = ... bool initOk = myClass.Initialize(); var result = myClass.PerformSomeAction(); Assert.IsTrue(initOk); Assert.AreEqual(42, result); }

37. [TestMethod] public void TestPasswordComplexity() { var result = _UserManager.ChangePassword(_TestUser.Id, "Password123!", "1!"); Assert.IsFalse(result.Succeeded); result = _UserManager.ChangePassword(_TestUser.Id, "Password123!", " "); result = _UserManager.ChangePassword(_TestUser.Id, "Password123!", " !"); result = _UserManager.ChangePassword(_TestUser.Id, "Password123!", "abcdefghijk"); result = _UserManager.ChangePassword(_TestUser.Id, "Password123!", "abcdefghijK1!"); Assert.IsTrue(result.Succeeded); }

38. Sometimes multiple asserts make sense
[TestMethod] public void CompareTwoAsserts() { var actual = GetNextMessage(); Assert.AreEqual(1, actual.Id); Assert.AreEqual("str-1", actual.Content); }

39. The multiple assertion dilemma
Multiple asserts
Lose information
Complex tests
One assert per test
Writing many tests for the same scenario
See Above

40. public class AssertAll { public static void Execute(params Action[] assertionsToRun) { var errorMessages = new List<exception>(); foreach (var action in assertionsToRun) { try { action.Invoke(); } catch (Exception exc) errorMessages.Add(exc); } if(errorMessages.Any()) var separator = string.Format("{0}{0}", Environment.NewLine); string errorMessage = string.Join(separator, errorMessages); Assert.Fail(string.Format("The following conditions failed:{1}", errorMessage)); } }

41. Why tests throw exceptions on failure?
It’s trivial, and best practice
Helps Decouple test runner from test
But not necessary…

42. Some frameworks are catching up!

43. This is excellent (JUnit 5)
@Test void groupedAssertions() {
assertAll("person",
() -> assertEquals("John", person.getFirstName()),
() -> assertEquals("Doe", person.getLastName()) ); }

44. This is Going too far (JUnit 5)
@Test void dependentAssertions() { assertAll("properties", () -> { String firstName = person.getFirstName(); assertNotNull(firstName); // Executed only if the previous assertion is valid. assertAll("first name", () -> assertTrue(firstName.startsWith("J")), () -> assertTrue(firstName.endsWith("n"))); }, // Grouped assertion, so processed independently of results of first name assertions. String lastName = person.getLastName(); assertNotNull(lastName); // Executed only if the previous assertion is valid. assertAll("last name", () -> assertTrue(lastName.startsWith("D")), () -> assertTrue(lastName.endsWith("e"))); }); }
This is Going too far (JUnit 5)

45. I wish my unit testing framework would:
Help me Write maintainable tests
Show me the root cause of a test failure
Point out which tests I forgot to write
Warn about memory leaks & threading issues
Make my work easier!
We’re Still missing guidance  more innovation is required!

46. TEST_CASE(“Total price equals sum of added products”) { FakeRepository fake_repository; Product product1 = Product(1, "product-1", 100); Product product2 = Product(2, "product-12, 200); EXPECT_CALL(fake_repository, GetProductById(1)).WillRepeatedly(Return(product1)); EXPECT_CALL(fake_repository, GetProductById(2)).WillRepeatedly(Return(product2)); ShoppingCart shopping_cart(fake_repository); shopping_cart.AddProductById(1); shopping_cart.AddProductById(2); REQUIRE(shopping_cart.GetTotalPrice() == 300); }

47. TEST_CASE(“Total price equals sum of added products”) { FakeRepository fake_repository; Product product1 = Product(1, "product-1", 100); Product product2 = Product(2, "product-2, 200); EXPECT_CALL(fake_repository, GetProductById(1)).WillRepeatedly(Return(product1)); EXPECT_CALL(fake_repository, GetProductById(2)).WillRepeatedly(Return(product2)); SECTION(“Add two products  return sum of products”) { ShoppingCart shopping_cart(fake_repository); shopping_cart.AddProductById(1); shopping_cart.AddProductById(2); REQUIRE(shopping_cart.GetTotalPrice() == 300); }

48. Dror Helper | @ dhelper | http://helpercode.com
Thank you
Dror Helper dhelper |