1. 1 Phase Testing

2. Janice Regan, 2008 2 For each group of units Overview of Implementation phase Create Class Skeletons Define Implementation Plan (+ determine subphases) Define Coding Standards For each unit Implement Methods in class/es Code review Unit test Create integration Test plan Create unit Test plans Release unit for integration Integration Testing System Testing Create system Test plan

3. Janice Regan, 2008 3 Testing  Goal: Deliberately trying to cause failures in a software system in order to detect any defects that might be present  Test effectively: uncovers as many defects as possible  Test efficiently: find largest possible number of defects using fewest possible tests

4. Janice Regan, 2008 4 Selection ofTest Case  Selection of test cases based on two strategies:  Black box testing strategy  Select test cases by examining expected behaviour and results of software system  Glass (white) box testing strategy  Select test cases by examining internal structure of code

5. Janice Regan, 2008 5 Select Set of Test Cases using Glass (White) Box Testing Strategy Goal: Select test cases, i.e., determine values of input, that will cause flow of execution to exercise as much (greatest coverage) of your method as possible  Test as much of your method as possible with the minimum number of test cases.  Compromise between extent of test coverage and time needed to complete the tests  More robust testing → longer test phase → more expensive testing  Less robust testing → shorter test phase → less expensive testing

6. Janice Regan, 2008 6 Glass (White) Box Testing  We treat the software system as a glass box  Can see inside, i.e., we knows about  Source code (steps taken by algorithms)  Internal data  design documentation  We can select test cases using our knowledge of the course code, hence test cases can exercise all aspects of each algorithm and data  More thorough but more time-consuming than black box testing

7. Janice Regan, 2008 7 Glass (White) Box Testing  Focus: Thoroughness (Coverage)  Test as much as is feasible  Base test design on looking in at the actual implementation  Various types of glass-box testing e.g.:  Statement testing  Branch testing  Path testing

8. Janice Regan, 2008 8 Glass (White) Box Testing  Statement testing:  Test all statements at least once  Branch Testing:  Make sure that each possible outcome from a condition is tested at least once  Path testing:  Make sure a fair # of paths in the program are executed at least once  Regarding loops: choose cases that will  Cause execution of the loop to be skipped completely  Cause loop to be executed exactly once  Cause loop to be executed more than once

9. Janice Regan, 2008 9  Select test cases (i.e., input values) that will cause each statement in your method to be executed at least once.  When performed (when your test cases are executed), your testing will achieve “100% statement coverage”.  Can miss problems when your method contains selection statements and loops Statement coverage

10. Janice Regan, 2008 10  Useful if your method contains selection statements  Select test cases (i.e., input values) that will cause each branch in your method to be executed at least once (if your method has selection statement(s), i.e., if statement(s)).  When performed (when your test cases are executed), your testing will achieve “100% branch coverage”  Can miss problems when the method contains loops Branch coverage

11. Janice Regan, 2008 11 Select test cases (i.e., input values) that will cause each path in your method to be executed at least once if your method has iterative statement(s), i.e., loop(s)). We will see in this lecture that it is not feasible to perform testing that achieve 100% path coverage (way too many test cases). To make up for what could be missed when path coverage testing is performed, Black-box testing strategy is also used to select additional test cases Often, Black-box testing strategy is also used to select additional test cases no matter which type of coverage is considered Path coverage

12. Janice Regan, 2008 12 1. Determine what to test which method of which class 2. Create a flowchart Break the code into blocks Represent the blocks in your flowchart 3.Depending on the type of code your method contains, selecting test cases to satisfy your goal can be done in three different ways: Procedure

13. Janice Regan, 2008 13 Glass-Box Testing Example /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); }

14. Janice Regan, 2008 14 Glass-Box Testing: choosing code blocks /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } 9 8 7 6 4 3 2 1 Where 5 corresponds to the empty “else” statement of the “if” statement # 3. Need for 5

15. Janice Regan, 2008 15 Finding test cases: Statement coverage  Find a minimal set of test cases that satisfy statement coverage  For statement coverage each line of the code must be executed.  If each line is executed all paths through the logic diagram will be taken  To get started, examine the method an break the code into logical blocks or units.  Then create a flowchart, showing the flow of execution between blocks

16. Janice Regan, 2008 16 Flowchart: Statement coverage 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Corresponding flowchart for statement coverage Each number on the diagram represents a corresponding block of code F

17. Janice Regan, 2008 17  What test cases (input values) would be necessary to achieve statement coverage for this example?  We are finding the mean of a series of positive numbers. The simplest case would be to find the of one positive number (the number itself). We would need an input file "input1" containing 1 positive number. Find Test cases: Statement coverage

18. Janice Regan, 2008 18 Test case1S: 1 data value (> 0) in file 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c, f d e e, i g h j k l F d Lower case letters show order of execution

19. Janice Regan, 2008 19  What additional test cases (input values) would be necessary to achieve 100% statement coverage for this example?  We are finding the mean of a series of positive numbers. The path that has not been executed is executed only if the sum of all numbers included in the mean is <=0. Since all numbers in the mean are positive this means that no numbers are included in the mean  Therefore, also need a test case using an empty input file "input2" as input Find Test cases: Statement coverage

20. Janice Regan, 2008 20 Test case 2S: Empty data file 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e F Lower case letters show order of execution Paths traversed shown as dotted lines

21. Janice Regan, 2008 21 Statement Coverage  Test case id – 1S  coverage: Start-1-2-3-4-6-2-7-8-Exit  Inputs: score file containing 1 score > 0.0  Test case id – 2S  coverage: Start-1-2-7-9-Exit  Inputs: empty score file  So, to achieve 100% statement coverage (test all statements), we would need two test cases, i.e., two set of input values

22. Janice Regan, 2008 22 Both test cases: all statements covered 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c, g f, j h i k l m e F Paths traversed shown as red or dotted black lines All lines of code are covered, Not all paths are covered

23. Janice Regan, 2008 23 Test Case Example  Test id - 1S  Test purpose: Unit test function FindMean( ) using Glass Box testing strategy (statement coverage: Start - 1 - 2 - 3 - 4 - 6 - 2 - 7 - 8 - Exit) (Note that FindMean( ) is a function, i.e., not a method of a class. If we are testing a method, we would state the class that is under testing as well.)  Requirement # 567  Inputs: score file “score” containing 1 score, value = 34.7 (> 0.0)  Testing procedure:  Open score file “score.dat” the call FindMean( … ) (part of unit test driver)  Evaluation: no step required  Expected behaviours and results: output “The mean score is 34.7” displayed on screen  Actual behaviours and results:

24. Janice Regan, 2008 24  Selecting our test cases based on statement coverage will not allow us to achieve 100% coverage when performing testing because  our function contains selection statements (branches, ifs).  Selecting test cases based on statement coverage may result in a set of test cases that miss some critical aspect of our function associated with the branches it includes. So…  What type of test cases (input values) could we come up with if we were to base our selection of test cases on covering branches of our function (not just statements) Are we done?

25. Janice Regan, 2008 25 Glass-Box Testing Example: Branches 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } Corresponding flowchart Each number on the diagram represents a corresponding block of code Each letter labels a particular branch of execution a b Data file must contain at least 1 value c e negative or 0 score g Positive score d f Total score > 0 i k h j Total score <= 0 l (Reached whether flow of control follows branch d (then f) or branch e (then g) ) (Data set must now be empty )

26. Janice Regan, 2008 26 Test case1B 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

27. Janice Regan, 2008 27 Branch Coverage  Test case id – 1B  coverage:  Start-a-b-d-f-h-b-e-g-h-c-i-k-Exit  Inputs: score file containing 2 scores  score 1 > 0.0  score 2 <= 0.0  We are finding the mean of a series of positive numbers. The path that has not been executed is executed only if the sum of all numbers included in the mean is <=0. Since all numbers in the mean are positive this means that no numbers are included in the mean  Therefore, also need a test case using an empty input file "input2" as input

28. Janice Regan, 2008 28 Test case 2B: Empty data file 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e f g h ij k l

29. Janice Regan, 2008 29 Branch Coverage  Test case id – 2B  coverage:  Start-a-c-j-l-Exit  Inputs: score file containing 0 scores

30. Janice Regan, 2008 30  Selecting our test cases based on branch coverage will not allow us to achieve 100% coverage when performing testing  our test cases based on branch coverage will result in a set of test cases that would still miss any potential error that could occur when the loop is executed many times (e.g. 13 times). So…  Let see what type of test cases (input values) we could come up with if we were to base our selection of test cases on covering (testing) paths of our function

31. Janice Regan, 2008 31 Glass-Box Testing: choosing code blocks /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } 9 8 7 6 4 3 2 1 Where 5 corresponds to the empty “else” statement of the “if” statement # 3. Need for 5

32. Janice Regan, 2008 32 Path Coverage  Test case id – 1P -> do while loop 0 time  coverage: Start-1-2-7-9-Exit  Test case id – 2P -> do loop once  Inputs: "score1P": a score file with 1 score > 0.0  coverage: Start-1-2-3-4-6-2-7-8-Exit  Test case id – 3P -> do loop once  coverage: Start-1-2-3-5-6-2-7-9-Exit  Inputs: "score1N: a score file with 1 score <= 0.0

33. Janice Regan, 2008 33 Test case 1P: Empty data file 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } a b c d e f g h ij k l

34. Janice Regan, 2008 34 Test case 2P 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

35. Janice Regan, 2008 35 Test case 3P 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

36. Janice Regan, 2008 36 Path Coverage  Test case id – 4P -> do loop many (3) times  one possible coverage: Start-1-2-3-4-6-2-3-5-6-2-3-4-6-2-7- 8-Exit  Inputs: "score3": a score file with 3 scores  score 1 > 0.0  score 2 <= 0.0  score 3 > 0.0

37. Janice Regan, 2008 37 Path Coverage  Test case id – 5P -> do loop many (13) times  one possible coverage: Start-1-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3- 4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6- 2-3-4-6-2-3-5-6-2-3-4-6-2-7-8-Exit  Inputs: "score13": a score file with 13 scores  score 1 > 0.0  score 2 > 0.0  …  score 12 <= 0.0  score 13 > 0.0  Test case id – etc…

38. Janice Regan, 2008 38 Test case 4P, 5P 9 8 7 6 4 3 2 1 /*Read in and sum the scores*/ FindMean( FILE scoreFile ) { float sumOfScores = 0.0; int numberOfScores = 0; float mean = 0.0; float score; /* Read score from file */ Read( ScoreFile, Score ); /* Total and count scores */ while ( !EOF(scoreFile ) ) { if ( score > 0.0 ) { sumOfScores = sumOfScores + score; numberOfScores++; } Read( scoreFile, score ); } /* Compute the mean and print the result */ if ( numberOfScores > 0 ) { mean = sumOfScores / numberOfScores; printf( "The mean score is %f \n", mean ); } else printf( "No scores found in file\n" ); } j a b e g h d f c i k l

39. Janice Regan, 2008 39  Considering the various paths the flow of execution can take in our function is the most extensive (closest to 100%) coverage we can achieve in our testing of FindMean( )  And this makes total sense considering the types of statements we find in your function  Therefore, since our function is composed of iterative statements, considering the paths the flow of execution can take when selecting our test cases (i.e., input values) is bound to give us the best coverage when we perform our testing of this function

40. Janice Regan, 2008 40 Redundant test cases  When you have selected your set of test cases (here, the set of test cases using path coverage), ensure that there are no redundant test cases within the set  Want the smallest set of test cases that gives adequate testing coverage  Avoid multiple test cases with the same coverage

41. Janice Regan, 2008 41 Coverage of proposed tests  1S -- coverage: Start-1-2-3-4-6-2-7-8-Exit  2S -- coverage: Start-1-2-7-9-Exit  1B -- coverage: Start-a-b-d-f-h-b-e-g-h-c-i-k-Exit  2B -- coverage: Start-a-c-j-l-Exit  1P -- coverage: Start-1-2-7-9-Exit  2P -- coverage: Start-1-2-3-4-6-2-7-8-Exit  3P -- coverage: Start-1-2-3-5-6-2-7-9-Exit  4P -- coverage: Start-1-2-3-4-6-2-3-5-6-2-3-4-6- 2-7-8-Exit  5P -- Start-1-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6-2- 3-4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6- 2-3-4-6-2-3-4-6-2-3-5-6-2-3-4-6-2-7-8-Exit

42. Janice Regan, 2008 42 Set of proposed tests  1P -- coverage: Start-1-2-7-9-Exit  2P -- coverage: Start-1-2-3-4-6-2-7-8-Exit  3P -- coverage: Start-1-2-3-5-6-2-7-9-Exit  4P -- coverage: Start-1-2-3-4-6-2-3-5-6-2- 3-4-6-2-7-8-Exit  And perhaps  5P -- Start-1-2-3-4-6-2-3-4-6-2-3-4-6-2-3- 4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3-4-6-2-3- 4-6-2-3-4-6-2-3-4-6-2-3-5-6-2-3-4-6-2-7- 8-Exit