1. CS 3773 Software Engineering Basic Concepts of Software Testing Unit Testing

2. 2 Why testing? Examples  Mars Climate Orbiter ($165M, 1998)  Sent to Mars to relay signal from the Mars Lander  Smashed to the planet: failing to convert between English measure to metric values  Shooting down of Airbus A300 (290 death, 1988)  US CG-49 shoot down a Airbus A300  Misleading output of the tracking software  THERAC-25 Radiation Therapy (1985)  2 cancer patients at East Texas Cancer center received fatal overdoses  Miss-handling of race condition of the software in the equipment

3. 3 Why testing? Numbers  On average, 1-5 bugs per KLOC (thousand lines of code)  In mature software  More than 10 bugs in prototypes  Windows2000  35MLOC  63K known bugs at the time of release  2 bugs per KLOC  $59.5B loss due to bugs in US 2002 (estimation by NIST)  It is not feasible to remove all bugs  But try to reduce critical bugs

4. 4 Answer is testing, why?  “50% of my employees are testers, and the rest spends 50% of their time testing” ---- Bill Gates, 1995  More reliable than inspection, relatively cheap  Actually in the old days, when testing is expensive, inspection was the major answer  You get what you pay (linear rewards)  Compared to other 3 approaches  Inspection, static checking, formal proof

5. 5 Testing: Concepts  Test case  Test oracle  Test suite  Test script  Test driver  Test result  Test coverage

6. 6 Testing: Concepts  Test case  An execution of the software with a given list of input values  Include:  Input values, sometimes fed in different steps  Expected outputs  Test oracle  The expected outputs of software by feeding in a list of input values  A part of test cases  Hardest problem in auto-testing: test oracle problem

7. 7 Testing: Concepts  Test suite  A collection of test cases  Usually these test cases share similar pre-requisites and configuration  Usually can be run together in sequence  Different test suites for different purposes  Smoke test, Certain platforms, Certain feature, performance, …  Test Script  A script to run a sequence of test cases or a test suite automatically

8. 8 Testing: Concepts  Test Driver  A software framework that can load a collection of test cases or a test suite  It can usually handle the configuration and comparison between expected outputs and actual outputs  Test Coverage  A measurement to evaluate how well the testing is done  The measure can be based on multiple elements  Code  Input combinations  Specifications

9. 9 Granularity of Testing: V-model

10. 10 Granularity of testing  Unit Testing  Test of a single module  Integration Testing  Test the interaction between modules  System Testing  Test the system as a whole, by developers on test cases  Acceptance Testing  Validate the system against user requirements, by customers with no formal test cases  Regression Testing  Test a new version with old test cases

11. 11 Unit testing  Testing of an basic module of the software  A function, a class, a component  Typical problems revealed  Local data structures  Algorithms  Boundary conditions  Error handling

12. 12 An example: w riting test cases without a test framework  Class Definition  Consider a class that handles and manages a Fibonacci array  Initialized the class  Fibonacci f = new Fibonacci ();  Extend to a certain length  f. extend (length)  Provide the number at certain index  f. get(index)  Get Length  f.getLength()  Provide a range of numbers  f. getRange(start, end)

13. 13 An example: writing automatic test cases without a test framework public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } … } Not Enough! What happened if an exception is thrown

14. 14 An example: writing automatic test cases without a test framework public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 50){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } … } Potential test interference!

15. 15 Common Operations  Comparison between actual results and expected results  Report error messages  Catch Exceptions  Re-initialize the class to be tested

16. 16 Unit test framework  xUnit  Created by Kent Beck in 1989  This is the same guy we mentioned in XP, design patterns  The first one was sUnit (for smalltalk)  Junit  The most popular xUnit framework  There are about 70 xUnit frameworks for corresponding languages  Never in the annals of software engineering was so much owed by so many to so few lines of code -------Martin Fowler

17. 17 State machine of JUnit Initialization Of Test Class setUp try{ testMethod } catch {…} tearDown Why? Try and Catch?

18. 18 Writing a Test Case public class FibonacciTest extends TestCase { protected Fibonacci fTest; protected static int groudtruth[] = {0, 1,...,12586269025}; public void setUp(){ fTest = new Fibonacci(); } @Test public testInitialize(){ assertEquals(“Length of fTest after initialization”, 0, fTest.getLength()); } @Test public testExtend(){ fTest.extend(50); assertEquals(“…”, 50, fTest.getLength()); } @Test public testGet(){ fTest.extend(50); assertEquals(“…”, groudtruth[49], fTest.get(50)); } … }

19. 19 Benefits of using test framework  Clear structure of testing  Each test method for one feature  Common initialization to setup  Assertion methods  You can always define new assertion methods  Reduced Interference  Try catch to run all test methods  Always do re-initialization in setup

20. 20 Demo  Writing a JUnit testcase for a class in your project  Run it as a JUnit test case

21. 21 Tips of writing test cases  Bottom Up  If you know that obj.m1 -> obj.m2, test obj.m2 first  Why?  You may not understand the error in m1 if you don’t know whether m2 is working or not  Externalize data  Externalize expected results to files and load them at the test case constructor  Why?  Easier to maintain if you want to reuse them in several places

22. 22 Tips of writing test cases  Use development Database  Do not use real database for testing  Why? Obvious  Do enough logging in the code during testing  Why?  Help debugging  Testing for exceptions  How? public testException(){ try{ do(invalid); fail(); }catch(xxxException e){} }

23. 23 Assertions public void testCapacity() { // a test method …. assertTrue(fFull.size() == 100+size); //assertion } If assertion fails: Assertion failed: myTest.java:150 (expected true but was false) Not so good! Try: assertEquals(100+size, fFull.size()); //expected value first Assertion failed: myTest.java:150 (expected 102 but was 103) Better! Try: assertEquals(“list length”, 100+size, fFull.size()); Assertion failed: myTest.java:150 (list length expected 102 but was 103)

24. 24 Assertions  Extend TestCase and write your own assertions  AssertStringContains, AssertArrayEquals, …  Use fail(String) to fail a test with certain message  Feel free to change the fixture  Each test will re-setup public void testRemoveAll() { fFull.removeAllElements(); fEmpty.removeAllElements(); assertTrue(fFull.isEmpty()); assertTrue(fEmpty.isEmpty()); }

25. Tear down  Consider the following test code void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { use f and b } void testBBB(){ use f and b } Problems? void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { try { use f and b } finally { clean&close f, b } void testBBB() { try { use f and b } finally { clean&close f, b } Better?

26. 26 Tear down  Consider the following test code void setUp() { File f = open(“foo”); File b = open(“bar”); } void testAAA() { use f and b } void testBBB(){ use f and b } void tearDown{ clean&close f, b } Problems? void setUp() { File f = open(“foo”); File b = open(“bar”); } … void tearDown{ try{ clean&close f }catch{ … } the same for b }

27. 27 Tear down  Be careful about tear down  If tear down is not complete, a test failure may affect the following test cases  Recover the changes done to global data that are not well handled by the setup  Database, files, network, global variables  Clean resources  Caution of exceptions in tear down itself