1. Technion – Institute of Technology 236700 Author: Gal Lalouche ©
Testing
Author: Gal Lalouche - Technion 2015 ©
Technion – Institute of Technology
236700
Author: Gal Lalouche ©

2. Why (unit) test? Programming is incremental by nature
We want to verify we haven’t broken anything
Tests not only examine the code’s functionality but also its
Ease of use
API
Boilerplate & setup
Error path behavior
Coherency and Coupling issues
Tests are the usage example of our code and are an excellent source for documentation
Author: Gal Lalouche - Technion 2015 ©

3. Testable Code → Good Code
Tests are usually the first time outside code will use your production code
Is the production code hard to use?
Is object creation long and complex?
Are methods cryptic with lots of parameters?
How does your production code handle usage errors?
Have you been surprised by a method’s behavior?
Is the production code hard to test?
Do you write LOC before you reach the assert line?
Is it difficult to assert the desired behavior?
Do you find it hard isolating the tested feature?
Are you not even sure what you should be asserting?
Author: Gal Lalouche - Technion 2015 ©
The most important reason to write tests is the improvement in the tested code design

4. Test types Unit Tests – Test a single element of the system
Our focus today
Unit Tests – Test a single element of the system
Integration Tests – Verify that a few units can work together and are wired up correctly
End-to-End Tests – Test the complete flow of interactions throughout the system
No “mocks”/”stubs”/”fakes”
Acceptance / Business logic Tests – Answers the question: “Did we build what we wanted?”
Usually centered around a user story
Author: Gal Lalouche - Technion 2015 ©
Single elements does not imply a single class
In the next tutorial we will discuss how to avoid writing integration tests when testing elements that are dependent on other elements

5. Test types
Author: Gal Lalouche - Technion 2015 ©

6. Testing Framework – Wish list
Better Asserts: Validating results Vs. expectations
assert(sqrt(4) == 2) Vs. assertEqual(2,sqrt(4))
assert(x != null) Vs. assertNotNull(x)
Clear indication of Success / Failure
Meaningful message upon failure
Ease of use – Should be easy to add new Tests
Aggregate tests together into Test Cases / Tests Suites
Continue with other tests even if a test fails / throws an exception
No duplication of initialization code of tested objects
Side effects should not influence other tests
Author: Gal Lalouche - Technion 2015 ©

7. JUnit A unit testing framework for Java A Test is a Method
A Test is a Method
A Test Case is a class which includes tests methods
Aggregates tests for a single tested unit
A Test Suite is a collection of Test Cases / Test Suites (or both)
Used to create a hierarchy of tests
Author: Gal Lalouche - Technion 2015 ©

8. JUnit 4.0 A step up from JUnit 3.0
New Assertions methods in org.junit.Assert.*
No need to extend from junit.framework.TestCase
No need to prefix test methods with ‘test’
Test methods are annotated
@BeforeClass – runs once before first test
Setting up the stubs objects / DB Connections
@AfterClass – runs once after the last test
Closing DB connections
@Before method – runs before every test
Preparing data
Could be used for advanced construction
@After method – runs after every test
Resetting stubs / cleaning data
Author: Gal Lalouche - Technion 2015 ©

9. JUnit test case life-cycle
@A* - Running all methods annotated with A
Test Case Class load
Static C’tor
@BeforeClass*
C’tor
@Before*
@Test
@After*
C’tor
@Before*
@Test
@After*
C’tor
@Before*
@Test
@After*
C’tor
@Before*
@Test
@After*
Author: Gal Lalouche - Technion 2015 ©
Note that the constructor is called before every test; use before for more complicated initializations
Avoid using multiple methods with the same lifecycle annotation (before, after, etc.)
@AfterClass*

10. JUnit life-cycle - Example
Local resources should be initialized and cleaned before/after every test
Author: Gal Lalouche - Technion 2015 ©
@BeforeClass must be static

11. JUnit life-cycle - Example
Expensive and global resources can be initialized and destroyed before/after all tests
Author: Gal Lalouche - Technion 2015 ©

12. Name is testing scenario
JUnit Example
Assuming we want to test a new Stack interface implementation
Smart Asserts:
assertNotEqual, assertTrue, assertSame, assertNotNull, assertArrayEqual, etc.
Name is testing scenario
Initialization
Author: Gal Lalouche - Technion 2015 ©
Naming is important!
Don’t just name it testPeek
Expectation
Result
org.junit.ComparisonFailure: expected:<[LAST]-ITEM> but was:<[FIRST]-ITEM>

13. Ideal Test A recipe for a good test:
If (1) and (2) are needed at all (e.g., initialization in it will be as short as possible (ideally just calls to new)
(3) will be a single method invocation
(4) will be a single assert
Of course this ideal cannot always be achieved
But if you can’t, you should always ask yourselves why?
@Test
public void testShouldNameExplicitlyStatesExpectation() {
(1) – Initialize collaborators (optional)
(2) – Initialize object under test (optional)
(3) – Exercise behavior
(4) – Assert outputs }
Author: Gal Lalouche - Technion 2015 ©
Keeping this test structure guarantees:
The test examines a single behavior
When the test fails It will be easy pinpointing the root cause
Good production code characteristics (mainly coherence)

14. Specifies the name of the invoked method but not the desired behavior
Test names – bad
It should be easy pin-pointing the problem when a test fails without looking at its code
Consider the following test:
How about addItemWithOutOfStockProduct?
Describes the method and the parameter
Still not a good name – does not describes the desired behavior
Bad name.
Specifies the name of the invoked method but not the desired behavior
@Test
public void addItemTest() {
  Product p = new Product();
  p.setAvailable(false);
  ShoppingCart cart = new ShoppingCart();
  cart.addItem(p);
  assertEquals(0, p.numItems()); }
Author: Gal Lalouche - Technion 2015 ©

15. Test names – good
It should be trivial understanding what is the expected behavior from the name alone
We do not need to mention the name of the invoked method
Good rule of thumb: the test name should include an explicit expectation
Use verbs such as should, does, returns, throws, etc.
@Test
public void doNotCountOutOfStockItem() {
  Product p = new Product();
  p.setAvailable(false);
  ShoppingCart cart = new ShoppingCart();
  cart.addItem(p);
  assertEquals(0, p.numItems()); }
Author: Gal Lalouche - Technion 2015 ©

16. JUnit – Exceptions A test that throws an exception will fail.
So how do we make sure a code throws an exception when it should?
We would like to verify 3 things:
The right exception has been thrown
The right line has thrown the exception (sometimes)
The exception contains the right data (rarely)
Author: Gal Lalouche - Technion 2015 ©
Do we always care about the message?

17. JUnit – Exceptions The hard way – Using the try/catch pattern
 Verifies our 3 wishes
 Easy to forget the fail resulting in a test that never fails
 Quite long and verbose
Test code is just as important as our “regular” code! We want to minimize code duplication and boilerplate.
@Test
public void badEncodeArgument() {
Encoder encoder = new Encoder(“UTF-8”);
try {
encoder.encode(null);
fail(“Exception should have been thrown”);
} catch (NullPointerException e) {
assertEquals(“Text should not be null”, e.getMessage()); }
Author: Gal Lalouche - Technion 2015 ©
Test code is just as important as our regular code! We want to minimize
Based on

18. JUnit – Exceptions
The easy way (JUnit 4.0) – expected value annotation
 Can’t check the exception message
 Can’t verify which line has thrown the exception
 Very short
Can be “good enough” in most cases
@Test(expected = IllegalArgumentException.class)
public void badEncodeArgument() {
Encoder encoder = new Encoder(“UTF-8”);
encoder.encode(null); }
Author: Gal Lalouche - Technion 2015 ©
Based on

19. JUnit – Exceptions
The smart way (JUnit 4.7) – annotation and ExpectedException
 Verifies our 3 wishes
 Still quite short
JUnit Rules offers a few additional candies
@Rule public ExpectedException thrown = ExceptedException.none();
@Test
public void badEncodeArgument() {
Encoder encoder = new Encoder(“UTF-8”);
thrown.expect(NullPointerException.class);
thrown.expectMessage(“Text should not be null”);
encoder.encode(null); }
Author: Gal Lalouche - Technion 2015 ©
Do we need to reset rule after every test? why/why not?
Based on

20. JUnit - Timeout
We want to verify that our code finishes in the allotted time.
This is criminally verbose
@Test
public void testTimeout() {
final boolean[] done = new boolean[] { false };
new Timer().schedule(new TimerTask() {
@Override public void run() {
foo();
done[0] = true; }
}, 0);
Thread.sleep(10);
if (done[0] == false)
fail(“Failed to run in the allotted time");
Author: Gal Lalouche - Technion 2015 ©
Why do we have to use an array?
Why can’t we use lambdas?
If foo throws an exception we have to catch it in the runnable, making the code even more verbose

21. JUnit - Timeout In JUnit 4.0 – Use the timeout annotation value
Test will fail after 10 milliseconds.
Global timeouts (JUnit 4.7)
@Test(timeout = 10)
public void testTimeout() {
// Tested code }
Author: Gal Lalouche - Technion 2015 ©
@Rule public MethodRule globalTimeout = new Timeout(10);
@Test
public void test1() {
// Tested code }
public void test2() {
Defines a timeout constraint for every test

22. JUnit – Ignoring tests
You can ignore tests using annotation
Test will not run, but will still be visible in the test reports
Commented out tests will usually be forgotten about
Won’t be deleted and possibly will not be fixed
Ignored tests are easily visible in the output window
Make sure you disable tests temporarily
Bad tests should either be deleted or fixed
Ask yourself – why are you ignoring the test in the first place?
Author: Gal Lalouche - Technion 2015 ©

23. JUnit - Running Textual runner IDE Support
Used mainly in command line environments (Scripts, Maven, Ant, CI)
IDE Support
To run several test cases
org.junit.runner.JUnitCore.runClasses(TestCase1.class, ...);
Author: Gal Lalouche - Technion 2015 ©
@RunWith(Suite.class)
@Suite.SuiteClasses({
TestCase1.class,
TestCase2.class,
subpackage.AllTests.class,
…})
public class AllTests {}
We can see that we have a hierarchy of test suites.
We can run only a sub-tree of the tests
We have one ignored test

24. JUnit – Source code Organization
Option 1: Test class in same folder as subject
A single hierarchy of folders
Locality
Option 2: Keep tests in the mirrored directory structure of source location
A duplicated hierarchy of folders
Easy to find all tests
Easy to separate source code from its tests
Easier bytecode separation
Author: Gal Lalouche - Technion 2015 ©
Maven default is option #2, but it can also be configured differently

25. Testing – Leveraging OOP
Tests are classes too!
Can use inheritance to our advantage
Use Abstract Test classes against abstract classes / interfaces
Classes
Tests Class A
ATest
Author: Gal Lalouche - Technion 2015 © B C
BTest
CTest

26. Testing – Leveraging OOP
public abstract class Stack<T> {
public void push(T element);
public int size(); }
public class ArrayStack<T> extends Stack<T> {
// … implementation code
public class LinkedListStackStack<T> extends Stack<T> {
Author: Gal Lalouche - Technion 2015 ©

27. Testing – Leveraging OOP
public abstract class StackTest {
private final Stack<Object> stack;
protected StackTest(Stack<Object> stack) {
this.stack = stack; }
@Test
public void sizeIsOneAfterOnePush() {
stack.push(new Object());
Assert.assertEquals(1, stack.size());
Author: Gal Lalouche - Technion 2015 ©

28. Testing – Leveraging OOP
Extend abstract TC to initialize the concrete test subjects
public class ArrayStackTest extends StackTest {
public ArrayStackTest() {
super(new ArrayStack<Object>()); }
// specific ArrayStack tests
Author: Gal Lalouche - Technion 2015 ©
public class LinkedListStackTest extends StackTest {
public LinkedListStackTest() {
super(new LinkedListStack<Object>()); }
// specific LinkedListStackTest tests

29. Testing – Order (or lack thereof)
Do not assume order between tests
There’s no order guaranties in TestCases
There is one in TestSuites
Avoid side effects in tests
public class StackTest {
private static Stack<Object> s = new Stack<>();
@Test
public void pushFirstAndCheckSize() {
s.push(new Object());
assertEquals(1,s.size()); }
public void pushSecondAndCheckSize() {
assertEquals(2,s.size());
Author: Gal Lalouche - Technion 2015 ©
Bad !

30. Testing – Resource Management
Do not load data from hard-coded locations on a file system
Use Class.getResource() or Class.getResourceAsStream()
Will search the file relatively to the Test Case .class file
Maven will automatically create a src & test resource folders
@Before
public void setUp () {
InputStream inp = FileInputStream("C:\\TestData\\dataSet1.dat"); … }
@Before
public void setUp () {
InputStream inp = getClass().getResourceAsStream("dataSet1.dat"); … }
Author: Gal Lalouche - Technion 2015 ©
“Do not load data from hard-coded locations on a file system” why not?
This works great with maven’s resource folders

31. Unit Testing: What to Test
It is not practical to test everything:
State space > 2 #𝑎𝑟𝑔𝑢𝑚𝑒𝑛𝑡𝑠+#𝑚𝑒𝑚𝑏𝑒𝑟𝑠 ⇒ Not always practical
Loops in the code ⇒ State space is infinite / Halting Problem
End to End and Acceptance tests will cover the state space quite well
Cover as much of the code as possible, but be smart about it
Extensiveness of the test should be proportional to the complexity of the implementation of the tested method
If m2() calls m1(), no need to duplicate m1() tests for m2()
But, for each variation in the flow (usually an if()) you should have a corresponding test that exercises it
Always test public methods
Should you test package-private and protected methods as well?
Private methods usage is important only in the context public methods
If you find yourself wanting to test a private method, you should rethink your design, (e.g., extract the private method to a helper class)
Author: Gal Lalouche - Technion 2015 ©

32. Unit Testing: Quality Cover code usage Code Coverage
Check common usage errors / edge cases:
nulls
empty strings
zeros
empty lists
single-item lists
uninitialized objects
bad configurations
Exception handling
Code Coverage
Function, Statement, Branch, Decision coverage
Monitor using external Code Coverage tools (e.g. EclEmma)
Author: Gal Lalouche - Technion 2015 ©

33. Code Coverage - EclEmma
Author: Gal Lalouche - Technion 2015 ©
A plugin for eclipse

34. Code Coverage
Code coverage is correlated with good testing, but does not equate it
You can achieve very high coverage with bad tests (e.g., testing without asserts)
You can write great tests with low coverage
100% coverage may look “pretty”, but should not be a goal in and off itself
You should always suspect tests that were written solely for achieving high coverage
Do you really need to test for NullPointerException for each of your parameters?
You can use code coverage to Spartanize your code
e.g., branches that are never taken may be dead code
Author: Gal Lalouche - Technion 2015 ©

35. Testing – Do’s & Don’ts Mutability makes code harder to test
Tests also have to take state into account
Always place shortest, simplest tests first in a TestCase or in a TestSuite
You should always make sure the simplest cases work before you carry on to the complicated ones
Prefer lots of short tests instead of a few long ones
Every test should test a single behavior
Speed up discovery-to-fix cycle
Testing code should be simpler than the subject code
Complicated tests usually implies complicated production code usage
Tests are the code usage examples
Do not put if statements in your tests. Loops are quite rare.
Author: Gal Lalouche - Technion 2015 ©

36. Testing – Do’s & Don’ts
Always make sure you start Green before you start coding
This way, you know if you broke something
Unless working in TDD…
In which case, start Green before refactoring!
Regression testing
Make sure new additions don’t break existing code
Run tests as often as possible
The earlier you detect the bug the sooner you solve it and the easier it becomes
Therefore, test runs should be short, at most 10ms per test
Never commit code that doesn’t pass all tests!
Author: Gal Lalouche - Technion 2015 ©

37. Testing – Do’s & Don’ts You can actually debug using unit tests!
When you work on a bug, write a test case first
Your work is done when the test case succeeds
If you write the test after the bug is fixed, you cannot know if the test succeeded in reproducing the bug
Helps with regression testing, i.e., ensuring the bug won’t reappear later
You can actually debug using unit tests!
Write a test that fails
If you still aren’t sure what the bug is, write a smaller unit test that fails
Repeat step 2 as necessary
This could be preferable to running a debugger, as the tests remain when done
Author: Gal Lalouche - Technion 2015 ©

38. Testing – Do’s & Don’ts Hard to test? ⇒ Re-think the design!
When you change the subject class, add tests
Testing is meant to improve the quality of the code
Do not write bad code to pass tests !!
Although possible in TDD…
Do not write bad tests to pass tests !! (Make sure a test can fail)
Hard to test? ⇒ Re-think the design!
Author: Gal Lalouche - Technion 2015 ©

39. Test types – misc. (not taught this semester)
Behavior-driven development
Using Domain Specific Language (DSL) to structure tests around stories and features
Property tests
Large randomized input to check invariants and properties of output
Mutability testing
Changing (mutating) the source code or tests slightly, to see if our tests fail
This ensures tight coupling between tests and source code
Design by Contract
Asserting pre and post-conditions on class’s methods
Fails in runtime, possible to check even at compile time
Author: Gal Lalouche - Technion 2015 ©
Single elements does not imply a single class
In the next tutorial we will discuss how to avoid writing integration tests when testing elements that are dependent on other elements