1. SE 433/333 Software Testing & Quality Assurance Dennis Mumaugh, Instructor dmumaugh@cdm.depaul.edu Office: CDM, Room 428 Office Hours: Tuesday, 4:00 – 5:30 April 19, 2016SE 433: Lecture 41 of 101

2. Administrivia  Comments and feedback  Announcements  Assignment 1 both parts have been graded.  Solution to Assignment 1 and 2 have been posted to D2L  Hints  Look at the Java documentation. For example: http://docs.oracle.com/javase/8/docs/api/java/lang/Double.html http://docs.oracle.com/javase/8/docs/api/java/lang/Double.html  Look at the examples (JUnit2.zip) mentioned on the reading list and provided in D2L.  In solving a problem, try getting the example working first. April 19, 2016SE 433: Lecture 4 2 of 101

3. SE 433 – Class 4 Topic:  Black Box Testing, JUnit Part 2 Reading:  Pezze and Young: Chapters 9-10  JUnit documentation: http://junit.org JUnit documentation: http://junit.org  An example of parameterized test: JUnit2.zip in D2L  See also reading list See also reading list April 19, 2016SE 433: Lecture 4 3 of 101

4. Assignments 4 and 5 Assignment 4: Parameterized Test  The objective of this assignment is to develop a parameterized JUnit test and run JUnit tests using Eclipse IDE.  Due Date: April 26, 2016, 11:59pm Assignment 5: Black Box Testing – Part 1: Test Case Design  The objective of this assignment is to design test suites using black- box techniques to adequately test the programs specified below. You may use any combination of black-box testing techniques to design the test cases.  Due date: May 3, 2016, 11:59pm April 19, 2016SE 433: Lecture 4 4 of 101

5. Thought for the Day “More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.” – Boris Beizer April 19, 2016SE 433: Lecture 45 of 101

6. Assignment 1 Lessons Learned  The assignment requirements may have some unwritten items:  Being able to test the program  Programs need to “defend” themselves from  Illegal input  Boundary conditions April 19, 2016SE 433: Lecture 4 6 of 101

7. Assignment 1 Part I, Grading rubric Part I: 10 Points  Compiles and runs and handles standard tests  Handles invalid cases -1  Not a triangle (a + b < c)  Degenerate triangle (a + b = c), aka line  Handles illegal input -1  Negative numbers  Floating point numbers  Extra large integers and MAXINT [See above triangle test].  Text  If the code handles most of the special condition give it the point. April 19, 2016SE 433: Lecture 4 7 of 101

8. Assignment 1 Part I, Grading rubric  For triangles:  10 + 20 = 30 hence it is a line  12 + 8 < 30 hence it is not a triangle  Negative length triangles are bad.  Also triangles with long numbers such as 2147483647. This number is 2^31 - 1 ) i.e. MAX_VALUE ”A constant holding the maximum value an int can have, 2 31 -1”. »The problem is in testing for illegal triangles one must check a + b <= c and we get arithmetic overflows. [See example on next slide.] »I got bit the first time myself.  The problem in using 21474836471 is simply that it is too large April 19, 2016SE 433: Lecture 4 8 of 101

9. Assignment 1 Lessons Learned  Consider: public class Triangle { public static void main(String[] args) { int a = 2147483647; int b = 2147483647; int c = 2147483647; if (a + b <= c) // Check a c b { System.out.println("Not a triangle"); } else { System.out.println("Triangle");} } Result: Not a triangle April 19, 2016SE 433: Lecture 4 9 of 101

10. Case Study – Knight Capital High Frequency Trading (HFT) April 19, 2016SE 433: Lecture 410 of 101

11. Case Study – Knight Capital: High Frequency Trading (HFT)  The Knight Capital Group is an American global financial services firm.  Its high-frequency trading algorithms Knight was the largest trader in U.S. equities  with a market share of 17.3% on NYSE and 16.9% on NASDAQ. April 19, 2016SE 433: Lecture 4 11 of 101

12. Case Study – Knight Capital  Aug 1, 2012.  In the first 45 minutes, Knight Capital's computers executed a series of unusually large automatic orders. “… spit out duplicate buy and sell orders, jamming the market with high volumes of trades that caused the wild swings in stock prices.”  By the end of day: $460 million loss  “Trading Program Ran Amok, With No ‘Off’ Switch”  In two days, the company's market value plunged by 75% April 19, 2016SE 433: Lecture 4 12 of 101

13. Case Study – Knight Capital: What Happened?  "Zombie Software" Blamed for Knight Capital Trading Snafu  A new algorithmic trading program had just been installed, and began operation on Aug 1.  A dormant legacy program was somehow "inadvertently reactivated"  Once activated, the dormant system started multiplying stock trades by one thousand  Sent 4 million orders when attempting to fill just 212 customer orders  “Knight’s staff looked through eight sets of software before determining what happened.” April 19, 2016SE 433: Lecture 4 13 of 101

14. Case Study – Knight Capital: The Investigation and Findings  SEC launched an investigation in Nov 2012. Findings:  Code changes in 2005 introduced defects. Although the defective function was not meant to be used, it was kept in.  New code deployed in late July 2012. The defective function was triggered under new rules. Unable to recognize when orders have been filled.  Ignored system generated warning emails.  Inadequate controls and procedures for code deployment and testing.  Charges filed in Oct 2013  Knights Capital settled charges for $12 million April 19, 2016SE 433: Lecture 4 14 of 101

15. Regression Test April 19, 2016SE 433: Lecture 415 of 101

16. Software Evolution  Change happens throughout the software development life cycle.  Before and after delivery  Change can happen to every aspect of software  Changes can affect unchanged areas »break code, introduce new bugs »uncover previous unknown bugs »reintroduce old bugs April 19, 2016SE 433: Lecture 4 16 of 101

17. Regression Test  Testing of a previously tested program following modification to ensure that new defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made.  It should be performed whenever the software or its environment is changed.  It applies to testing at all levels. April 19, 2016SE 433: Lecture 4 17 of 101

18. Regression Test  Keep a test suite  Use the test suite after every change  Compare output with previous tests  Understand all changes  If new tests are needed, add to the test suite. April 19, 2016SE 433: Lecture 4 18 of 101

19. Test Driven Development (TDD) April 19, 2016SE 433: Lecture 419 of 101

20. Test Early  Testing should start as early as possible  design test cases  Test early has several advantages  independence from design & code  discover inconsistencies and incompleteness of the specifications  serve as a compendium of the specifications April 19, 2016SE 433: Lecture 4 20 of 101

21. Test Driven Development  Test driven development (TDD) is one of the corner stones of agile software development  Agile, iterative, incremental development  Small iterations, a few units  Verification and validation carried out for each iteration.  Design & implement test cases before implementing the functionality  Run automated regression test of whole system continuously April 19, 2016SE 433: Lecture 4 21 of 101

22. Process of Test Driven Development  Tests should be written first (before any code)  Execute all test cases => all fail  Implement some functions  Execute all test cases => some pass  Repeat implement and re-execute all test cases  Until all test cases => pass  Refactoring, to improve design & implementation  re-execute all test cases => all pass  Every time changes are made  re-execute all test cases => all pass April 19, 2016SE 433: Lecture 4 22 of 101

23. Black Box Testing April 19, 2016SE 433: Lecture 423 of 101

24. Black Box View  The system is viewed as a black box  Provide some input  Observe the output April 19, 2016SE 433: Lecture 4 24 of 101

25. Functional Testing: A.k.a.: Black Box Testing  Derive test cases from the functional specifications  functional refers to the source of information  not to what is tested  Also known as:  specification-based testing (from specifications)  black-box testing (no view of the code)  Functional specification = description of intended program behavior  either formal or informal April 19, 2016SE 433: Lecture 4 25 of 101

26. Systematic vs. Random Testing  Random (uniform) testing  Pick possible inputs randomly and uniformly  Avoids designer bias  But treats all inputs as equally valuable  Systematic (non-uniform) testing  Select inputs that are especially valuable  Choose representatives  Black box testing is systematic testing April 19, 2016SE 433: Lecture 4 26 of 101

27. Why Not Random Testing?  Non-uniform distribution of defects  Example:  Program: solve a quadratic equation: a x 2 + b x + c = 0  Defect: incomplete implementation logic does not properly handle special cases: b 2 - 4ac = 0 and a = 0  Failing values are sparse in the input space — needles in a very big haystack.  Random sampling is unlikely to choose a=0.0 and b=0.0 April 19, 2016SE 433: Lecture 4 27 of 101

28. Systematic Partition of Input Space Failure No failure Failures are sparse in the space of possible inputs...... but dense in some parts of the space If we systematically test some cases from each part, we will include the dense parts Functional testing is one way of drawing lines to isolate regions with likely failures The space of possible input values April 19, 2016SE 433: Lecture 4 28 of 101

29. The Partition Principle  Exploit knowledge in problem domain to choose samples for testing  Focus on “special” or trouble-prone regions of the input space  Failures are sparse in the whole input space... ... but we may find regions in which they are dense  (Quasi*-) Partition testing  Separates the input space into classes whose union is the entire space  *Quasi because the classes may overlap April 19, 2016SE 433: Lecture 4 29 of 101

30. The Partition Principle  Desirable case for partitioning  Input values that lead to failures are dense (easy to find) in some classes of input space  Sampling each class in the quasi-partition by selecting at least one input value that leads to a failure, revealing the defect  Seldom guaranteed, depend on experience- based heuristics April 19, 2016SE 433: Lecture 4 30 of 101

31. Black Box Testing Exploiting the functional specification  Uses the specification to partition the input space  e.g., specification of “roots” program suggests division between cases with zero, one, and two real roots  Test each partition, and boundaries between partitions  No guarantees, but experience suggests failures often lie at the boundaries (as in the “roots” program) April 19, 2016SE 433: Lecture 4 31 of 101

32. Why Black Box Testing?  Early.  can start before code is written  Effective.  find some classes of defects, e.g., missing logic  Widely applicable  any description of program behavior as spec  at any level of granularity, from module to system testing.  Economical  less expensive than structural (white box) testing The base-line technique for designing test cases April 19, 2016SE 433: Lecture 4 32 of 101

33. Early Black Box Testing  Program code is not necessary  Only a description of intended behavior is needed  Even incomplete and informal specifications can be used »Although precise, complete specifications lead to better test suites  Early test design has side benefits  Often reveals ambiguities and inconsistency in spec  Useful for assessing testability »And improving test schedule and budget by improving spec  Useful explanation of specification »or in the extreme case (as in XP), test cases are the spec April 19, 2016SE 433: Lecture 4 33 of 101

34. Functional versus Structural: Classes of faults  Different testing strategies (functional, structural, fault- based, model-based) are most effective for different classes of faults  Functional testing is best for missing logic faults  A common problem: Some program logic was simply forgotten  Structural (code-based) testing will never focus on code that isn’t there! April 19, 2016SE 433: Lecture 4 34 of 101

35. Functional vs. Structural Test  Functional test is applicable in testing at all granularity levels:  Unit test(from module interface spec)  Integration test(from API or subsystem spec)  System test (from system requirements spec)  Regression test(from system requirements + bug history)  Structural test is applicable in testing relatively small parts of a system:  Unit test April 19, 2016SE 433: Lecture 4 35 of 101

36. Steps: From specification to test cases 1. Decompose the specification If the specification is large, break it into independently testable features to be considered in testing 2. Select representatives  Representative values of each input, or  Representative behaviors of a model Often simple input/output transformations don’t describe a system. We use models in program specification, in program design, and in test design 3. Form test specifications Typically: combinations of input values, or model behaviors 4. Produce and execute actual tests April 19, 2016SE 433: Lecture 4 36 of 101

37. From specification to test cases April 19, 2016SE 433: Lecture 4 37 of 101

38. An Example: Postal Code Lookup  Input: ZIP code (5-digit US Postal code)  Output: List of cities What are some representative values to test? April 19, 2016SE 433: Lecture 4 38 of 101

39. Example: Representative Values  Correct zip code  With 0, 1, or many cities  Malformed zip code  Empty; 1-4 characters; 6 characters; very long  Non-digit characters  Non-character data Simple example with one input, one output Note prevalence of boundary values (0 cities, 6 characters) and error cases April 19, 2016SE 433: Lecture 4 39 of 101

40. Summary  Functional testing, i.e., generation of test cases from specifications is a valuable and flexible approach to software testing  Applicable from very early system specs right through module specifications  (quasi-)Partition testing suggests dividing the input space into (quasi-)equivalent classes  Systematic testing is intentionally non-uniform to address special cases, error conditions, and other small places  Dividing a big haystack into small, hopefully uniform piles where the needles might be concentrated April 19, 2016SE 433: Lecture 4 40 of 101

41. Basic Techniques of Black Box Testing April 19, 2016SE 433: Lecture 441 of 101

42. Single Defect Assumption Failures are rarely the result of the simultaneous effects of two (or more) defects. April 19, 2016SE 433: Lecture 4 42 of 101

43. Functional Testing Concepts The four key concepts in functional testing are:  Precisely identify the domain of each input and each output variable  Select values from the data domain of each variable having important properties  Consider combinations of special values from different input domains to design test cases  Consider input values such that the program under test produces special values from the domains of the output variables April 19, 2016SE 433: Lecture 4 43 of 101

44. Developing Test Cases  Consider: Test cases for input box accepting numbers between 1 and 1000  If you are testing for an input box accepting numbers from 1 to 1000 then there is no use in writing thousand test cases for all 1000 valid input numbers plus other test cases for invalid data.  Using equivalence partitioning method, above test cases can be divided into three sets of input data called as classes. Each test case is a representative of respective class.  We can divide our test cases into three equivalence classes of some valid and invalid inputs. April 19, 2016SE 433: Lecture 4 44 of 101

45. Developing Test Cases 1. One input data class with all valid inputs. Pick a single value from range 1 to 1000 as a valid test case. If you select other values between 1 and 1000 then result is going to be same. So one test case for valid input data should be sufficient. 2. Input data class with all values below lower limit. I.e. any value below 1, as a invalid input data test case. 3. Input data with any value greater than 1000 to represent third invalid input class. So using equivalence partitioning you have categorized all possible test cases into three classes. Test cases with other values from any class should give you the same result. April 19, 2016SE 433: Lecture 4 45 of 101

46. Equivalence Classes  Equivalence classes are the sets of values in a (quasi-) partition of the input, or output domain  Values in an equivalence class cause the program to behave in a similar way:  failure or success  Motivation:  gain a sense of complete testing and avoid redundancy  First determine the boundaries … then determine the equivalencies April 19, 2016SE 433: Lecture 4 46 of 101

47. Determining Equivalence Classes  Look for ranges of numbers or values  Look for memberships in groups  Some may be based on time  Include invalid inputs  Look for internal boundaries  Don’t worry if they overlap with each other —  better to be redundant than to miss something  However, test cases will easily overlap with boundary value test cases April 19, 2016SE 433: Lecture 4 47 of 101

48. Selecting Data Points  Determining equivalence classes for each input variable or field  Single input variable  Normal test »Select one data point from each valid equivalence class  Robustness test »Include invalid equivalence class April 19, 2016SE 433: Lecture 4 48 of 101

49. Selecting Data Points  Multiple input variables  Weak normal test: »Select one data point from each valid equivalence class  Strong normal test: »Select one data point from each combination of (the cross product of) the valid equivalence classes  Weak/strong robustness test: »Include invalid equivalence classes  How many test cases do we need? April 19, 2016SE 433: Lecture 4 49 of 101

50. Example of Selecting Data Points  Suppose a program has 2 input variables, x and y  Suppose x can lie in 3 valid equivalence classes:  a ≤ x < b  b ≤ x < c  c ≤ x ≤ d  Suppose y can lie in 2 valid equivalence classes:  e ≤ y < f  f ≤ y ≤ g April 19, 2016SE 433: Lecture 4 50 of 101

51. Weak Normal Test  Every normal, i.e., valid, equivalence class of every input variable is tested in at least one test case.  A representative value of each normal equivalence class of each input variable appears in at least one test case.  Economical, requires few test cases if the values are selected prudently.  Complete. April 19, 2016SE 433: Lecture 4 51 of 101

52. Weak Normal Test April 19, 2016SE 433: Lecture 4 52 of 101

53. Strong Normal Test  Every combination of normal equivalence classes of every input variable is tested in at least one test cases.  More comprehensive.  Requires more test cases.  May not be practical for programs with large number of input variables. April 19, 2016SE 433: Lecture 4 53 of 101

54. Strong Normal Test a bc d y x e f g April 19, 2016SE 433: Lecture 4 54 of 101

55. Weak Robustness Test  Add robustness test cases to weak normal test suite.  Every invalid equivalence class of every input variable is tested in at least one robustness test case.  Each robustness test case include only one invalid input value.  No combination of invalid input values. April 19, 2016SE 433: Lecture 4 55 of 101

56. Weak Robustness Test April 19, 2016SE 433: Lecture 4 56 of 101

57. Strong Robustness Test  Add robustness test cases to strong normal test suite.  Every invalid equivalence class of an input variable is tested with all combinations of valid equivalence classes of other input variable.  Each robustness test case include only one invalid input value.  No combination of invalid input values. April 19, 2016SE 433: Lecture 4 57 of 101

58. Strong Robustness Test Cases a bc d y x e f g April 19, 2016SE 433: Lecture 4 58 of 101

59. Summary  For Multiple input variables  Weak normal test: »Select one data point from each valid equivalence class  Strong normal test: »Select one data point from each combination of (the cross product of) the valid equivalence classes  Weak/strong robustness test: »Include invalid equivalence classes April 19, 2016SE 433: Lecture 4 59 of 101

60. Example: nextDate() Function  This program reads a date in the format of mm/dd/yyyy and prints out the next date.  For example, an input of 03/31/2014 gives an output of 04/01/2014  A constraint (arbitrary, for illustration purpose only)  The year is between 1800 and 2200 inclusive April 19, 2016SE 433: Lecture 4 60 of 101

61. Example: nextDate(): Valid Equivalence Classes  The valid equivalence classes for the Day  { 1 ≤ Day ≤ 28 }  { Day = 29 }  { Day = 30 }  { Day = 31 }  The valid equivalence classes for the Month  { Month has 30 days }  { Month has 31 days }  { Month = February }  The valid equivalence classes for the Year  { Year is not a leap year }  { Year is a leap year } April 19, 2016SE 433: Lecture 4 61 of 101

62. Example: nextDate(): Invalid Equivalence Classes  The invalid equivalence classes for the Day { Day 31 } { Incorrect format of Day } { Illegal characters of Day }  The invalid equivalence classes for the Month { Month 12 } { Incorrect format of Month }{ Illegal characters of Month }  The invalid equivalence classes for the Year { Year 2200 } { Incorrect format of Year }{ Illegal characters of Year }  Other invalid equivalence classes { Incorrect order of Day, Month, Year } { Missing Day, Month, or Year } { Extra number or character } April 19, 2016SE 433: Lecture 4 62 of 101

63. Example: nextDate(): Test Cases: Weak Normal  Valid equivalence classes and data points  Day Data Points »{ 1 ≤ Day ≤ 28 } 10 »{ Day = 29 } 29 »{ Day = 30 }30 »{ Day = 31 }31  Month »{ Month has 30 days }04 »{ Month has 31 days } 03 »{ Month = February }02  Year »{ Year is not a leap year }2009 »{ Year is a leap year }2008  Weak normal test cases (4 cases) 1. 02/10/2009 2. 04/29/2009 3. 03/30/2008 4. 03/31/2008 April 19, 2016SE 433: Lecture 4 63 of 101

64. Example: nextDate(): Test Cases: Strong Normal  Strong normal test cases (17 cases) 02/10/2008 02/29/2008 02/10/2009 03/10/2008 03/29/2008 03/30/2008 03/31/2008 03/10/2009 03/29/2009 03/30/2009 03/31/2009 04/10/2008 04/29/2008 04/30/2008 04/10/2009 04/29/2009 04/30/2009  Note: some combinations are invalid, thus excluded  e.g., 02/30/2008 April 19, 2016SE 433: Lecture 4 64 of 101

65. Example: nextDate(): Test Cases: Weak Robustness  Add a test case for each invalid equivalence class  { Day < 1 }02/00/2008  { Day > 31 } 03/36/2009  { Incorrect format of Day } 02/7/2008  { Illegal characters of Day } 02/First/2008  { Month < 1 } 00/10/2009  { Month > 12 }15/10/2008  { Incorrect format of Month }3/10/2008  { Illegal characters of Month } Mar/10/2009  { Year < 1800 }02/10/1745  { Year > 2200 }02/10/2350  { Incorrect format of Year }02/10/10  { Illegal characters of Year } 02/10/’00  { Incorrect order of Day, Month, Year }29/03/2008  { Missing Day, Month, or Year }02/10  { Extra number or character }02/20/2008/2009 April 19, 2016SE 433: Lecture 4 65 of 101

66. Example: nextDate(): Test Cases: Strong Robustness  Add invalid test cases resulting from combination of valid equivalence classes 04/31/2008 02/29/200902/30/2009 02/31/2009 02/30/200802/31/2008  Ensure each invalid test case contains only one invalid value.  Single defect assumption April 19, 2016SE 433: Lecture 4 66 of 101

67. Boundary Value Testing  Test values, sizes, or quantities near the design limits  value limits, length limits, volume limits  null strings vs. empty strings  Errors tend to occur near the extreme values of inputs (off by one is an example)  Robustness:  How does the software react when boundaries are exceeded?  Use input values  at their minimum, just above the minimum  at a nominal value,  at the maximum, just below the maximum April 19, 2016SE 433: Lecture 4 67 of 101

68. Input Boundary Values  Test cases for a variable x, where a ≤ x ≤ b  Experience shows that errors occur more frequently for extreme values of a variable. a b x x(min)x(min+)x(nom)x(max -)x(max) April 19, 2016SE 433: Lecture 4 68 of 101

69. Input Boundary Values – 2 Variables Test cases for a variables x 1 and x 2, where a ≤ x 1 ≤ b and c ≤ x 2 ≤ d  Single defect assumption a b c d x 2 x 1 April 19, 2016SE 433: Lecture 4 69 of 101

70. Example: nextDate() – Test Cases: Boundary Values  Additional test cases, valid input 04/01/200904/30/2009 03/01/200903/31/2009 02/01/200902/28/2009 02/29/2008 01/01/200812/31/2008 01/01/180012/31/2200 April 19, 2016SE 433: Lecture 4 70 of 101

71. Robustness Testing  Test cases for a variable x, where a ≤ x ≤ b  Stress input boundaries  Acceptable response for invalid inputs?  Leads to exploratory testing (test hackers)  Can discover hidden functionality ab x April 19, 2016SE 433: Lecture 4 71 of 101

72. Robustness Testing – 2 Variables a b c d x 2 x 1 April 19, 2016SE 433: Lecture 4 72 of 101

73. Example: nextDate() – Test Cases: Boundary Values  Additional robustness test cases, invalid input 04/00/200904/31/2009 03/00/200903/32/2009 02/00/200902/29/2009 02/30/2008 01/00/200812/32/2008 12/31/179901/01/2201 00/01/200913/01/2009 April 19, 2016SE 433: Lecture 4 73 of 101

74. Worst-Case Testing  Discard the single-defect assumption  Worst-case boundary testing:  Allow the input values to simultaneously approach their boundaries  Worst-case robustness testing:  Allow the input values to simultaneously approach and exceed their boundaries April 19, 2016SE 433: Lecture 4 74 of 101

75. Worst Case Boundary Testing – 2 Variables a b c d x 2 x 1 April 19, 2016SE 433: Lecture 4 75 of 101

76. Worst Case Robustness Testing – 2 Variables a b c d x 2 x 1 April 19, 2016SE 433: Lecture 4 76 of 101

77. Limitations of Boundary Value Testing  Doesn’t require much thought  May miss internal boundaries  Usually assumes the variables are independent  Values at the boundary may not have meaning April 19, 2016SE 433: Lecture 4 77 of 101

78. Special Value Testing  The most widely practiced form of functional testing  The tester uses his or her domain knowledge, experience, or intuition to probe areas of probable errors  Other terms: “hacking”, “out-of-box testing”, “ad hoc testing”, “seat of the pants testing”, “guerilla testing” April 19, 2016SE 433: Lecture 4 78 of 101

79. Uses of Special Value Testing  Complex mathematical (or algorithmic) calculations  Worst case situations (similar to robustness)  Problematic situations from past experience  “Second guess” the likely implementation April 19, 2016SE 433: Lecture 4 79 of 101

80. Characteristics of Special Value Testing  Experience really helps  Frequently done by the customer or user  Defies measurement  Highly intuitive  Seldom repeatable  Often, very effective April 19, 2016SE 433: Lecture 4 80 of 101

81. Summary: Key Concepts  Black-box testing  vs. random testing, white-box testing  Partitioning principle  Black box testing techniques  Equivalence class  Boundary value testing  Special value testing  Single defect assumption  Normal vs. robustness testing  Weak and strong combinations April 19, 2016SE 433: Lecture 4 81 of 101

82. Guidelines and observations  Equivalence Class Testing is appropriate when input data is defined in terms of intervals and sets of discrete values.  Equivalence Class Testing is strengthened when combined with Boundary Value Testing  Strong equivalence takes the presumption that variables are independent. If that is not the case, redundant test cases may be generated April 19, 2016SE 433: Lecture 4 82 of 101

83. An Introduction to JUnit Part 2 April 19, 2016SE 433: Lecture 483 of 101

84. JUnit Best Practices  Each test case should be independent.  Test cases should be independent of execution order.  No dependencies on the state of previous tests. April 19, 2016SE 433: Lecture 4 84 of 101

85. JUnit Test Fixtures  The context in which a test case is executed.  Typically include:  Common objects or resources that are available for use by any test case.  Activities to manage these objects  Set-up: object and resource allocation  Tear-down: object and resource de-allocation April 19, 2016SE 433: Lecture 4 85 of 101

86. Set-Up  Tasks that must be done prior to each test case  Examples:  Create some objects to work with  Open a network connection  Open a file to read/write April 19, 2016SE 433: Lecture 4 86 of 101

87. Tear-Down  Tasks to clean up after execution of each test case.  Ensures  Resources are released  the system is in a known state for the next test case  Clean up should not be done at the end of a test case,  since a failure ends execution of a test case at that point April 19, 2016SE 433: Lecture 4 87 of 101

88. Method Annotations for Set-Up and Tear-Down  @Before annotation: set-up  code to run before each test case.  @After annotation: Teardown  code to run after each test case.  will run regardless of the verdict, even if exceptions are thrown in the test case or an assertion fails.  Multiple annotations are allowed  all methods annotated with @Before will be run before each test case  but no guarantee of execution order April 19, 2016SE 433: Lecture 4 88 of 101

89. Example: Using a File as a Test Fixture public class OutputTest { private File output; @Before public void createOutputFile() { output = new File(...); } @After public void deleteOutputFile() { output.close(); output.delete(); } @Test public void test1WithFile() { // code for test case … } @Test public void test2WithFile() { // code for test case … } April 19, 2016SE 433: Lecture 4 89 of 101

90. Method Execution Order 1. createOutputFile() 2. test1WithFile() 3. deleteOutputFile() 4. createOutputFile() 5. test2WithFile() 6. deleteOutputFile() Not guaranteed: test1WithFile runs before test2WithFile April 19, 2016SE 433: Lecture 4 90 of 101

91. Once-Only Set-Up  @BeforeClass annotation on a static method  one method only  Run the method once only for the entire test class  before any of the tests, and  before any @Before method(s)  Useful for starting servers, opening connections, etc.  No need to reset/restart for each test case  Shared, non-destructive @BeforeClass public static void anyName() { // class setup code here } April 19, 2016SE 433: Lecture 4 91 of 101

92. Once-Only Tear-Down  @ After Class annotation on a static method  one method only  Run the method once only for the entire test class  after any of the tests  after any @After method(s)  Useful for stopping servers, closing connections, etc. @AfterClass public static void anyName() { // class clean up code here } April 19, 2016SE 433: Lecture 4 92 of 101

93. Timed Tests  Useful for simple performance test  Network communication  Complex computation  The timeout parameter of @Test annotation  in milliseconds  The test fails  if timeout occurs before the test method completes @Test(timeout=5000) public void testLengthyOperation() {... } April 19, 2016SE 433: Lecture 4 93 of 101

94. Parameterized Tests  Repeat a test case multiple times with different data  Define a parameterized test  Class annotation, defines a test runner @RunWith(Parameterized.class)  Define a constructor »Input and expected output values for one data point  Define a static method returns a Collection of data points »Annotation @Parameter [or @Parameters, depending] »Each data point: an array, whose elements match the constructor arguments April 19, 2016SE 433: Lecture 4 94 of 101

95. Running a Parameterized Test  Use a parameterized test runner  For each data point provided by the parameter method  Construct an instance of the class with the data point  Execute all test methods defined in the class April 19, 2016SE 433: Lecture 4 95 of 101

96. Parameterized Test Example – Program Under Test public class Calculator { public long factorial(int n) { … return result; } public class Calculator { public long factorial(int n) { … return result; } See Junit2.zip April 19, 2016SE 433: Lecture 4 96 of 101

97. Parameterized Test Example – The Test Class @RunWith(Parameterized.class) public class CalculatorTest { private long expected; // expected output private int value; // input value public CalculatorTest(long expected, int value) { this.expected = expected; this.value = value; } @RunWith(Parameterized.class) public class CalculatorTest { private long expected; // expected output private int value; // input value public CalculatorTest(long expected, int value) { this.expected = expected; this.value = value; } April 19, 2016SE 433: Lecture 4 97 of 101

98. Parameterized Tests Example – The Parameter Method @Parameters public static Collection data() { return Arrays.asList(new Integer[][] { { 1, 0 }, // expected, value { 1, 1 }, { 2, 2 }, { 24, 4 }, { 5040, 7 }, }); } @Parameters public static Collection data() { return Arrays.asList(new Integer[][] { { 1, 0 }, // expected, value { 1, 1 }, { 2, 2 }, { 24, 4 }, { 5040, 7 }, }); } April 19, 2016SE 433: Lecture 4 98 of 101

99. Parameterized Tests Example – The Test Method private long expected; // expected output private int value; // input value …. @Test public void factorialTest() { Calculator calc = new Calculator(); assertEquals(expected, calc.factorial(value)); } private long expected; // expected output private int value; // input value …. @Test public void factorialTest() { Calculator calc = new Calculator(); assertEquals(expected, calc.factorial(value)); } April 19, 2016SE 433: Lecture 4 99 of 101

100. Readings and References  Chapter 10 of the textbook.  JUnit documentation  http://junit.org http://junit.org  An example of parameterized test  JUnit2.zip in D2L April 19, 2016SE 433: Lecture 4 100 of 101

101. Next Class Topic: Black Box Testing Part 2, JUnit & Ant Reading: Chapter 10 of the textbook. Articles on the class page and reading list Assignment 4 – Parameterized Test  Due April 26, 2016 Assignment 5 – Black Box Testing – Part 1: Test Case Design  Due May 3, 2016 April 19, 2016SE 433: Lecture 4 101 of 101