1. Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING White-Box Testing Claus Brabrand [ brabrand@itu.dk ] ( “FÅP”: First-year Project Course, ITU, Denmark )

2. [ 2 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Outline Motivation Why bother with testing? What is Testing? Relation to other programming-related tasks? White-box Testing Methodology: ”coverage testing” Automation How to automate testing Exercises Training for the exam

3. [ 3 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Learning & Exam Goals ”Product”: ”Oral Exam”: Today, we’ll ”train for the exam”… :-)

4. [ 4 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING  Software Errors Mobile Phones ’00-… Freeze and odd behaviors (really annoying)! Cruise Control System Model ’86 (Grady Booch) Accellerated after car ignition  car crashes Baggage Handling System ’94-’95 (at Denver Int’l Airport) $ 360,000,000 USD

5. [ 5 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING z z Software Errors (cont’d) Train Control System ’98 (Berlin) Train cancellations Mars Pathfinder July ’97 Periodic resets Win95/98 w/ 3 rd -Party Device Drivers late ’90es Dysfunction (“blue screen of death”)!...on mars!

6. [ 6 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Software Errors (cont’d 2 ) Therac-25 Radiation Therapy ’85-’87 Massive overdoses (6 deaths / amputations)! Patriot Missile Guidance System ’91 (Gulf War 1.0) Accumulating rounding errors  deaths Ariane V ’96 (one of the most expensive bugs, ever) Conversion from 64-bit float to 16-bit signed int

7. [ 7 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING  …and what about?! Surgical Laser Control System Oops…! Air Plane Control System Dysfunction (plane crash)! Nuclear Powerplant Control System Core melt-down (“China-syndrome”)!

8. [ 8 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Outline Motivation Why bother with testing? What is Testing? Relation to other programming-related tasks? White-box Testing Methodology: ”coverage testing” Automation How to automate testing Exercises Training for the exam

9. [ 9 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Errors! (different kinds) Syntactic errors: Mal-formed program: Semantic errors: Symbol errors Type errors Other semantic errors: (e.g. uninitialized vars) Logical errors: Compiler: ”no errors” int square(int x) { return x*x } *** syntax error at line 2 ’;’ expected int square(int x) { return n*n; } *** symbol error at line 2 undefined variable ”n” int square(float x) { return x*x; } *** type error at line 2 function returns float, not int int square(int x) { return x+x; } no errors found!!!

10. [ 10 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING ISO 9126 Maintainability Reliability Usability Portability How robust is the SW wrt. incorrect inputs, ’^C’, external netwk failures,...? How easy is the SW to understand? …and use? How easy is it to transfer and adapt SW to new environment / platform? How easy is it to modify the SW? And fix errors? Int’l standard for evaluation of: Software Quality Functionality Efficiency ISO 9126 Does the SW do what it’s supposed to? Does it work as intended? How much time/memory/space/- bandwidth/… does the SW consume?

11. [ 11 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Testing vs. Debugging? Testing vs. Debugging: Functionality:Efficiency: Quality Assurance: (Functionality) Testing (Performance) Testing Diagnosis:Profiling Purpose: Regarding: Debugging

12. [ 12 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Testing vs. Debugging (cont’d) Evaluate test results Fix problem (reprogram) Functionality Testing Quality assurance: Is program ok? Debugging Diagnosis: Determine problem? 0110 1021 Program: 0110 1011 (greater confidence!)   SYSTEMATIC Document test results (confidence?!?) Re- 0110 1011

13. [ 13 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Performance Testing vs. Profiling Evaluate test results Improve program (reprogram) Performance Testing Quality assurance: Efficient enough? Profiling Diagnosis: Determine problem? 0110 1021 Program: 0110 1011   (greater confidence!) Document test results (confidence?!?) Re- SYSTEMATIC 0110 1011

14. [ 14 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Testing…: ”Testing is easy” (e.g., ”random experimentation”) ”Testing well is not easy”  Requires SYSTEMATIC approach; test-case  production  evaluation  documentation ”Testing can never prove error absence” (i.e., testing is an ”incomplete process”)

15. [ 15 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Representative? Comprehensive? Quality? Quantity? …? Appropriate Test Sampling?    

16. [ 16 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING White-box vs. Black-box Test White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”) Test focus: source code Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”) Test focus: specification (manual) Complementary Approaches!!! n = in(); n = n/2; odd(n) n = 3*n+1; out(n); tt ff ~ ? programspec

17. [ 17 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING ”Software Testing” (R. Patton) Background reading: ”Software Testing”, Ron Patton, Sams Publishing, 2006 Part II (pp. 53 – 123); ”Testing Fundamentals”: ”Examining the Code” (chapter 6) ”Examining the Spec.” (chapter 4) ”Testing w/ Blinders On” (chapter 5) ”Testing w/ X-ray Glasses” (chapter 7) Dynamic (at runtime) Static (before runtime) -testing Time: Type:

18. [ 18 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Outline Motivation Why bother with testing? What is Testing? Relation to other programming-related tasks? White-box Testing Methodology: ”coverage testing” Automation How to automate testing Exercises Training for the exam

19. [ 19 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Test Coverage? Method coverage: Does every method run (at least once)? Statement coverage: Does every statement run (at least once)? Branch coverage: Does every branch run (at least once)? Path coverage: Does every path run (at least once)?

20. [ 20 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Statement coverage Branch coverage: Does every branch run (at least once)? -Box ”Branch Coverage Testing” is: Efficient (fast) ! Effective (thorough) ! Good for complicated program logic (esp. ”initialization errors”)

21. [ 21 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Control Structures Control Structures: Statements (or Expr’s) that affect ”flow of control”: if-else : if : if ( Exp ) Stm 1 else Stm 2 if ( Exp ) Stm Stm 1 Exp truefalse Stm 2 confluence Stm Exp true false confluence The expression must be of type boolean; if it evaluates to true, the given statement is executed, otherwise not. The expression must be of type boolean; if it evaluates to true, Statement-1 is executed, otherwise Statement-2 is executed. [syntax] [semantics] [syntax] [semantics]

22. [ 22 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Control Structures (cont’d) while : for : while ( Exp ) Stm for (Exp 1 ; Exp 2 ; Exp 3 ) Stm Equivalent to: The expression must be of type boolean; if it evaluates to false, the given statement is skipped, otherwise it is executed and afterwards the expression is evaluated again. If it is still true, the statement is executed again. This is continued until the expression evaluates to false. { Exp1; while ( Exp2 ) { Stm Exp3; } } Stm Exp true false confluence Exp 1 ; Exp 2 true false Stm confluence Exp 3 ; [syntax] [semantics] [syntax] [semantics]

23. [ 23 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Stm/Branch Coverage Testing if : TEST condition true and false if-else : TEST condition true and false while : TEST zero, one, more-than-one iterations in loop for : TEST zero, one, more-than-one iterations in loop

24. [ 24 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Example 1 public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma); }} /* 1 if-else */ /* 2 for */ /* 3 if-else */ /* 4 if */ if else for if else if Choice points?

25. [ 25 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Control-Flow Graph CFG: int mi, ma; System.out.println ("No numbers"); args.length == 0 mi = ma = Integer.parseInt(args[0]); int i=1; i < args.length i++; System.out.println(”min=" + mi + "," + "max=" + ma); int obs = Integer.parseInt(args[i]); obs > ma ma = obs; mi < obs mi = obs; true false true false true false true false 2 3 4 1

26. [ 26 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING ”Coverage Table”: Coverage Table Input property No numbers At least one number Exactly one number Exactly two numbers At least three numbers N > current max N  current max N  cur max & N > cur min N  cur max & N  cur min Data set A B B C E C D E (3 rd num) E (2 nd num) Choice 1 ife true false 2 for zero-times once more-than-once 3 ife true false 4 if true false

27. [ 27 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING ”Expectancy Table”: Expectancy Table Data set A B C D E      Advice: Avoid expected 0’s (i.e., zeroes) (Default value in many languages.) Advice: Avoid reusing same numbers in tests (Data layout sometimes reuse old memory.) Input  [17] [27,29] [39,37] [49,47,48] Expected output ”no numbers” ”min=17,max=17” ”min=27,max=29” ”min=37,max=39” ”min=47,max=49” Actual output ”no numbers” ”min=17,max=17” ”min=27,max=29” ”min=39,max=39” ”min=49,max=49” =

28. [ 28 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma); }} Debugging ’ D ’ then reveals… /* 1 if-else */ /* 2 for */ /* 3 if-else */ /* 4 if */ if else for if else if Should have been: (obs < mi)

29. [ 29 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Re-Test ! …as debugging often introduces new errors ! Fixed Program: Coverage Table: Expectancy Table:  Recall: no guarantee!

30. [ 30 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Example 2 public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1 if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3 if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4 for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6 if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2); } } }

31. [ 31 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Coverage Table (Ex. 2) Choice Input property Data set 1 ife true No numbers 1 ife false At least one number 2 ife true Exactly one number 2 ife false At least two numbers 3 if true 2 nd number ≥ 1 st number 3 if false 2 nd number < 1 st number 4 for zero-times Exactly two numbers 4 for once Exactly three numbers 4 for more-than-once At least four numbers 5 ife true 3 rd number < current min 5 ife false 3 rd number ≥ current min 6 if true 3 rd ≥ cur min & 3 rd < 2 nd least 6 if false 3 rd ≥ cur min & 3 rd ≥ 2 nd least ABBCCDDEHEFFGABBCCDDEHEFFG

32. [ 32 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Expectancy Table (Ex. 2) Data set Input Expected output = Actual output A  ”no numbers” ”no numbers” B [17] ”17” ”17” C [27,29] ”27 and 29” ”27 and 0” D [39,37] ”37 and 39” ”37 and 39” E [49,48,47] ”47 and 48” ”47 and 48” F [59,57,58] ”57 and 58” ”57 and 58” G [67,68,69] ”67 and 68” ”67 and 0” H [77,78,79,76] ”76 and 77” ”76 and 77”         Debugging reveals that variable ” mi2 ” erroneously retains initialization (0).

33. [ 33 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Debugging (Ex. 2) public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1 if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3 if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4 for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6 if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2); } } } mi2 = obs; Re-Test: 

34. [ 34 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Control Structures (cont’d 2 ) switch : do-while : ” ?: ”; ”conditional expression”: ” && ”; ”lazy conjunction” (aka., ”short-cut  ”): ” || ”; ”lazy disjunction” (aka., ”short-cut  ”): switch ( Exp ) { Swb * } case Exp : Stm * break; do Stm while ( Exp ); default : Stm * break; Swb: Exp 1 ? Exp 2 : Exp 3 Exp 1 && Exp 2 Exp 1 || Exp 2 Choice points?

35. [ 35 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Control Structures (cont’d 3 ) try-catch-finally (exceptions): return / break / continue : ”method invocation”: e.g.; ”recursive method invocation”: e.g.; ”virtual dispatching”: e.g.; try Stm 1 catch ( Exp ) Stm 2 finally Stm 3 f(x) return ;return Exp ; break ;continue ; f(x)

36. [ 36 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Outline Motivation Why bother with testing? What is Testing? Relation to other programming-related tasks? White-box Testing Methodology: ”coverage testing” Automation How to automate testing Exercises Training for the exam

37. [ 37 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Test Automation (Re-)Running tests is boooring (& error prone) Thus, automate them ”once-and-for-all” JUnit: Can be run from Eclipse/JUnit: (if appropriately ”subclassing” TestCase ) public class MyTestCase extends TestCase { /*...other tests... */ } @Test // Testing if 3*2=6: public void testMultiplication() { assertEquals("Multiplication", 6, 3*2); }

38. [ 38 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Outline Motivation Why bother with testing? What is Testing? Relation to other programming-related tasks? White-box Testing Methodology: ”coverage testing” Automation How to automate testing Exercises Training for the exam

39. [ 39 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Exercise: Part I: Part II:  Program merge (in Java):  Test your merge method: a)Label choice points b)Build ”coverage table” & make data set (test suite) c)Build ”expectancy table” d)Run test suite (upon failure: fix and retest program)  Introduce subtle bug  Run test to document presence of bug  Submit “erroneous program” to class program pool List merge(List list1, List list2);  Pick “erroneous program” from class program pool  Test merge program (and debug to find bug)  Re-Test fixed merge program  Write report (and send it to the teaching assistant) (produce) (consume) Warm up exercise: Draw a control-flow diagram for the ” do-while ” construction

40. [ 40 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Specification (merge) Interface (for ”merge”): I/O assumptions: Input: both lists are sorted: …and, numbers occur maximum once (in each list) Output: list must be sorted: …and, numbers occur maximum once Programming constraints: no recursion ! no java.util.Iterator ’s ! List merge(List list1, List list2);

41. [ 41 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Report (ca. 3 pages): Must explain how you tested, (debugged), & re-tested the erroneous ” merge ” (it must include, at least): i) The erroneous program ii) a ”Control-Flow Graph” for the program [hand-drawn ok] (incl. labelled ”choice points”) iii) ”Coverage Tables” iv) ”Expectancy Tables” Submit the testing report to the T.A. (Anders) (deadline: Tuesday; March, 11 at 09:00 CET) Note: the report is only on Part II (i.e., the program you didn’t write) The Testing Report

42. [ 42 ] Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING Example 1 public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma); }}