1. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Førsteårsprojekt (F2010) Claus Brabrand [ brabrand@itu.dk ] IT University of Copenhagen

2. [ 2 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Agenda (Feb 2, 2010) Introduction to the course & project Groups Surprise (educational technology) Introduction to testing Control-flow White-box testing

3. [ 3 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Danmarkskort (krak data) Danmarkskort: Visualisering Naviagation Søgning Ruteplanlægning... ”Danmarkskort: Visualisering, Navigation, Søgning og Ruteplanlægning”

4. [ 4 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Proces: Konstitution Konstitution: A constitution is a system for government, often codified as a written document, that establishes the rules and principles of an autonomous [political] entity.

5. [ 5 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Proces: Dokumenter 1) Personlige notater: alle tager notater! 2) Projekt-dagbog: normer, aftaler, idéer, møde-referater, beslutninger, beslutnings-grundlag og -form, eksterne samtaler 3) Arbejdsblade: til enhver tid sammenfatte fælles faglige resultater 4) Projekt-rapport: formidler projektets resultater (fra 'analyse' til 'konklusion') 5) Proces-beskrivelse: formidler erfaringer med + egen vurdering af arbejdsproces -- v2.1-- (grundlag for komm. med selv+gruppen) (grundlag for pæd. vejledning) (grundlag for faglig vejledning) (grundlag for evaluering af produkt) (grundlag for eval. af proces) weekly submission!

6. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Groups FAAP’09 Project Groups

7. [ 7 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Considerations (Group Formation) Considerations; e.g.: a) Balance Introverts and Extroverts: Too many intros  few discussions / limited exchange Too many extros  Blah blah blah… b) Avoid ”strategist-strategist” clashes: More likely to ”clash” c) Balance heterogeneously wrt. ”MBTI” Many different inputs d) Avoid ”old group mates” in same group Learn to work with ”new people” Avoid sub-groups (aka, cliques)

8. [ 8 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Groups (MBTI Types) ISTJ ( Inspector ) ISTP ( Crafter ) ESTP ( Promoter ) ESTJ ( Supervisor ) ISFJ ( Protector ) ISFP ( Composer ) ESFP ( Performer ) ESFJ ( Provider ) INFJ ( Counselor ) INFP ( Healer ) ENFP ( Champion ) ENFJ ( Teacher ) INTJ ( Mastermind ) INTP ( Architect ) ENTP ( Inventor ) ENTJ ( Fieldmarshal ) SN TFT I E P J J

9. [ 9 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White:  Tomas L.  Anders Ø.  Nikolaj Aa.  Nicolai S.  Ahmad S. Yellow:  Sune A.  Nicklas W.  Martin K.  Thomas D.  Marc F. Orange:  Liv T.  Martin D.  Jonas J.  Thorkil B.  Søren N. Red:  Christian V.  Jeppe T.  Christian H.  Anne D.P.  Mads C. Green:  Nicolai D.  Benjamin M.  Patrick A.  Nicolai T.  Niels J. Blue:  Jonathan R.  Anders M.  Lasse C.Aa.  Anders H.K.  Michael B.M Brown:  Signe M.H.  Thomas T.H  Hildur F.  Niklas J.  Asger S. Black:  Jesper R.  Kristian S.A.  Morten H.  Peter Ø.  Kristian S. Groups: Gray:  Mikkel H.  Jason S.  Christoffer A  Søren E.  Kasper A. Purple:  Emil B.M.  Mark A.  Mischa S.  Benjamin H.  Martin J.L.

10. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Clickers ! :-)

11. [ 11 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 What is the average rainfall of the Amazon basin? A. 20 mm/year B. 200 mm/year C. 2000 mm/year D. 20000 mm/year Copenhagen: 587 mm/year

12. [ 12 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 What is the fourth most populous nation in the world? A. Pakistan B. Brazil C. Indonesia D. Nigeria E. Russia TOP 3: China (1,338 mio) India (1,166 mio) USA (307 mio)

13. [ 13 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 What is the following expression: "A  B" logically equivalent to? A. A  B B.  A   B C.  (  A   B) D.  (A  B) E.  (  A   B) F.  (A  B)

14. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Introduction to Testing Claus Brabrand [ brabrand@itu.dk ] ( First-year Project Course, ITU, Denmark )

15. [ 15 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Outline Motivation: Why bother with testing? The Psychology of Testing: What is the goal of testing? The Testing Process: What is relation to other programming-related tasks? Bugs: Important properties of bugs Implementation vs. Specification: Formal definition of a bug

16. [ 16 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Learning & Exam Goals ”Product”: ”Oral Exam”:

17. [ 17 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Software Errors Therac-25 Radiation Therapy ’85-’87 Massive overdoses (6 deaths / amputations)! Patriot Missile Guidance System ’91 (Gulf War 1.0) Accumulating rounding errors  deaths Ariane V ’96 (one of the most expensive bugs, ever) Conversion from 64-bit float to 16-bit signed int

18. [ 18 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 z z Software Errors (cont’d) Train Control System ’98 (Berlin) Train cancellations Mars Pathfinder July ’97 Periodic resets Win95/98 w/ 3 rd -Party Device Drivers late ’90es Dysfunction (“blue screen of death”)!...on mars!

19. [ 19 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010  Software Errors (cont’d) Mobile Phones ’00-… Freeze and odd behaviors (really annoying)! Cruise Control System Model ’86 (Grady Booch) Accellerated after car ignition  car crashes Baggage Handling System ’94-’95 (at Denver Int’l Airport) $ 360,000,000 USD

20. [ 20 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010  …and what about?! Surgical Laser Control System Eye damage (blindness)! Air Plane Control System Dysfunction (plane crash)! Nuclear Powerplant Control System Core melt-down (“China-syndrome”)!

21. [ 21 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Outline Motivation: Why bother with testing? The Psychology of Testing: What is the goal of testing? The Testing Process: What is relation to other programming-related tasks? Bugs: Important properties of bugs Implementation vs. Specification: Formal definition of a bug

22. [ 22 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Testing: Incomplete Process A program has: many possible valid inputs many possible invalid inputs Hence: 8 8 Testing can never prove absence of errors! Testing is an incomplete process!

23. [ 23 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 The Psychology of Testing X Goal: test a program to "show that it works" Homo Sapiens are ”goal oriented” (steer towards goal)! …we might as well stop before we even start! :-( ”Testing is the process of demonstrating that errors are not present.” ”The purpose of testing is to show that a program performs its intended functions correctly.” [cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979] ”Testing is the process of establishing confidence that a program does what it is supposed to do.”

24. [ 24 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010   successful test case unsuccessful test case Problematic terminology:   successful test case unsuccessful test case Much better terminology: Psychology of Testing (cont’d) Goal: find as many errors as possible Note: realistically assumes errors are present Constructive goal (actually destructive) ”Testing is the process of executing a program with the intent of finding errors.” vs [cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979]

25. [ 25 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Constructive vs. Destructive Thinking Constructive thinking: (e.g., programming) ”Test-to-pass” Often not a good idea to ”test” your own code :-( Destructive thinking: (e.g., testing) ”Test-to-fail” Often better to test/break someone else’s code :-) Recommendation:  Have another group help you test/break your group ”product”

26. [ 26 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Tester’s Goal The goal of a software tester is: Testers (therefore) often aren’t the most popular members of a project team :-) - 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed” [R.Patton, “Software Testing”, p. 19] [cf. ”Software Testing”, R.Patton, p.19]

27. [ 27 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Outline Motivation: Why bother with testing? The Psychology of Testing: What is the goal of testing? The Testing Process: What is relation to other programming-related tasks? Bugs: Important properties of bugs Implementation vs. Specification: Formal definition of a bug

28. [ 28 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 ISO 9126 Maintainability Reliability Usability Portability How robust is the SW wrt. external network failures, ’^C’,...? How easy is the SW to understand? …and use? How easy is it to transfer and adapt SW to new environment / platform? How easy is it to modify the SW? And fix errors? Software Quality International evaluation standard Functionality Efficiency ISO 9126 Does the SW do what it’s supposed to? Does it work as intended? How much time/memory/space/- bandwidth/… does the SW consume?

29. [ 29 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Testing vs. Debugging? Testing vs. Debugging?: Functionality:Efficiency: Quality Assurance: (Functionality) Testing (Performance) Testing Diagnosis:Profiling Purpose: Regarding: Debugging

30. [ 30 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Testing vs. Debugging (cont’d) Evaluate test results Fix problem (reprogram) Functionality Testing Quality assurance: Perform (functionality) test Debugging Diagnosis: Determine problem? 0110 1021 Program: 0110 1011 (greater confidence!)   SYSTEMATIC Document test results (confidence?!?) Re- 0110 1011 2 1

31. [ 31 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Performance Testing vs. Profiling Evaluate test results Improve program (reprogram) Performance Testing Quality assurance: Perform (efficiency) test Profiling Diagnosis: Determine problem? 0110 1021 Program:   (confidence?!?) Re- SYSTEMATIC 0110 1011 (greater confidence!) Document test results

32. [ 32 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Testing vs. Debugging (cont’d) Evaluate test results Fix problem (reprogram) Functionality Testing Quality assurance: Perform (functionality) test Debugging Diagnosis: Determine problem? 0110 1021 Program: 0110 1011 (greater confidence!)   SYSTEMATIC Document test results (confidence?!?) Re- 0110 1011 2 1

33. [ 33 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White-box vs. Black-box Test White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”) Test focus: source code Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”) (aka., ”input-ouput testing” ) Test focus: specification (or intention) Complementary Approaches!!! n = in(); n = n/2; odd(n) n = 3*n+1; out(n); tt ff ~ ? programspec

34. [ 34 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Exercise: which is (usually) ”best” - and why? A) white-box testing ; black-box testing ; (i.e., white-box testing first) B) black-box testing ; white-box testing ; (i.e., black-box testing first) Answer: ’B’ Settle overall impl ~ spec problems first Before zooming in on details of impl Exercise …or…

35. [ 35 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Outline Motivation: Why bother with testing? The Psychology of Testing: What is the goal of testing? The Testing Process: What is relation to other programming-related tasks? Bugs: Important properties of bugs Implementation vs. Specification: Formal definition of a bug

36. [ 36 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Definition: ”bug” Main entry: 2 bug Pronunciation: /’bəg/ Function: noun Etymology: origin unknown Date: 1622 1 a: an insect or other creeping or crawling invertebrate (as a spider or centipede) b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious c: any of an order (Hemiptera and especially its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests —called also true bug 2: an unexpected defect, fault, flaw, or imperfection 3 a: a germ or microorganism especially when causing disease b: an unspecified or nonspecific sickness usually presumed due to a bug 4: a sudden enthusiasm 5: enthusiast 6: a prominent person 7: a crazy person 8: a concealed listening device 9: a weight allowance given apprentice jockeys

37. [ 37 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 ”The Harvard Mark II Bug” Photo of first actual ”bug”: “The first documented computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested” Harvard University, Sep. 9, 1947

38. [ 38 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Example of a Bug :-) Hej Claus, A propos test og datoberegninger: Fredag morgen mente min PDA kalender (Microsoft Windows CE) at vi skrev lørdag 1. marts. Jeg stillede selvfølgelig tålmodigt uret tilbage til fredag 29. februar 2008 (dette fandt sted på et møde hos Microsoft i Vedbæk ;-) Lørdag formiddag startede den så med at vise min (ret tomme) kalender for august 2049. Det indbyggede ur stod på 1. januar 1601. Ahem. Peter

39. [ 39 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Terminology (for ’Bugs’) ”Severe conditions”: Fault Failure Defect ”Unintended operation”: Anomaly Incident Variance Feature (sounds intended) ”Generic terms”: Problem Error Bug Typically imply blame; as in: - ”it was his fault” Famous quote: - ”It’s not a bug, it’s a feature” we’ll use these terms

40. [ 40 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Bugs Bugs often occur in groups: If you find one, chances are there are more nearby This property useful when testing program parts: i.e., figuring out where to concentrate testing efforts Not all bugs will be fixed: Too costly? Too risky? Not ”cost-effective”? Cause unknown?

41. [ 41 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Cost of (Fixing) Bugs Cost of bugs increases exponentially (over time): (log) specdesigncodetestrelease $1 $100 $10,000 $1,000,000 $100,000,000 [cf. ”Software Testing”, R.Patton, p.18] cost phase

42. [ 42 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 ”The Pesticide Paradox” ”The pesticide paradox”: ”The more you test a software, the more immune it becomes to your tests” B.Beizer, “Software Testing Techniques”, 1990

43. [ 43 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Different 'kinds' of errors Syntactic errors: Mal-formed program: Semantic errors: Symbol errors Type errors Other semantic errors: (e.g. uninitialized vars) Logical errors: Compiler: ”no errors” int square(int x) { return x*x } *** syntax error at line 2 ’;’ expected int square(int x) { return n*n; } *** symbol error at line 2 undefined variable ”n” int square(float x) { return x*x; } *** type error at line 2 function returns float, not int int square(int x) { return x+x; } no errors found!!!

44. [ 44 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Outline Motivation: Why bother with testing? The Psychology of Testing: What is the goal of testing? The Testing Process: What is relation to other programming-related tasks? Bugs: Important properties of bugs Implementation vs. Specification: Formal definition of a bug

45. [ 45 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Definition: ”Bug” A ”bug” is a relation between: Specification & Implementation Whether or not specification is: Explicit or Implicit Written or Oral Formal or Informal (e.g., ”product spec” vs. ”back-of-envellope”) implementation (aka., ’program’) specification (aka., ’spec’) ~ bug

46. [ 46 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Specification A spec states things an impl…: Should do! Shouldn’t do! Unspecified? (’unclear’, ’unmentioned’, or ’left open’) Should do! Shouldn’t do! Unspecified? S Specs are often intentionally under-specified. It’s often better to not ”prematurely commit” to a particular solution (by specifying exactly how a task should be done) – and instead just state which overall tasks the system should do.

47. [ 47 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Formal Definition: ”Bug” A (software) bug occurs when: 1. Impl doesn’t do sth spec says it should: 2. Impl does sth spec says it shouldn’t: 3. Impl does sth spec doesn’t mention: 4. Spec doesn’t mention sth, but should: 5. Impl is hard to understand/use by user(s): ~ ~ ?! (or does it incorrectly) I S I S S I ?! I S [cf. ”Software Testing”, R.Patton, p.15] ! ~ ideal

48. [ 48 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Which of the conditions (1-5) does this violate (if any)? A. It's not a bug B. Bug wrt. #1 C. Bug wrt. #2 D. Bug wrt. #3 E. Bug wrt. #4 F. Bug wrt. #5 ”Additional functionality”? e.g., a calculator which also does square root "  " (but wasn’t supposed to) ' '' '

49. [ 49 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Which of the conditions (1-5) does this violate (if any)? A. It's not a bug B. Bug wrt. #1 C. Bug wrt. #2 D. Bug wrt. #3 E. Bug wrt. #4 F. Bug wrt. #5 ”Easter egg”? e.g., hit [Alt+Shift+2] in Solitaire to instantly win game

50. [ 50 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Which of the conditions (1-5) does this violate (if any)? A. It's not a bug B. Bug wrt. #1 C. Bug wrt. #2 D. Bug wrt. #3 E. Bug wrt. #4 F. Bug wrt. #5 ”An (good) impl w/o a spec”? i.e., only a piece of software (no specification whatsoever)

51. [ 51 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Clicker Exercise (cont'd) Exercise: Ron Patton interprets as (”spec := impl”):  trivially no bugs in impl ! (according to [own] definition #1,#2,#3) More sensible to interpret as ”no spec”:  entire impl is essentially one big bug ! (…according to definition #3+#4) ”Because the software is already complete, you have the perfect specification – the product itself” [R.Patton, ”Software Testing”, p.31] ”an impl w/o a spec” ? bug ~ [cf. ”Software Testing”, R.Patton, p.15 v. p.31]

52. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Test Cases

53. [ 53 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Testing…: Testing is easy: (e.g., ”random experimentation”) Testing well is not easy; requires: A SYSTEMATIC approach Test case…: production evaluation documentation

54. [ 54 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Representativity! Appropriate Test Cases?     SYSTEMATIC TESTING !

55. [ 55 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Myers’ 10 Testing Principles 1) expected output is part of a test case 2) avoid testing own program 3) avoid testing own (organization’s) program 4) thoroughly inspect result of each test case 5) also write ”invalid / unexpected” test cases 6) also check: doesn’t do what not supposed to 7) avoid throw-away test cases 8) never assume no errors when making test cases 9) Prob(more errors) ~ Prob(#errors already found) 10) Testing is creative+intellectually challenging task [cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979] a) destructive vs. constructive mode b) may be overall misunderstandings instead: systematic ”regression testing”! a) … b) …

56. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Control-Flow Claus Brabrand [ brabrand@itu.dk ] ( First-year Project Course, ITU, Denmark )

57. [ 57 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White-box vs. Black-box Test White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”) Test focus: source code Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”) (aka., ”input-ouput testing” ) Test focus: specification (or intention) n = in(); n = n/2; odd(n) n = 3*n+1; out(n); tt ff ~ ? programspec

58. [ 58 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Test Coverage? Method coverage: Does every method run (at least once)? Statement coverage: Does every statement run (at least once)? Branch coverage: Does every branch run (at least once)? Path coverage: Does every path run (at least once)?

59. [ 59 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Statement coverage Statement coverage: Does every statement run (at least once)? -Box ”Statement Coverage Testing” is: Efficient (fast) ! Effective (thorough) ! Good for complicated program logic (esp. ”initialization errors”)

60. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Control Structures & Control Flow Graphs

61. [ 61 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Control Structures Control Structures: Statements (or Expr’s) that affect ”flow of control”: if-else : if : if ( Exp ) Stm 1 else Stm 2 if ( Exp ) Stm Stm 1 Exp truefalse Stm 2 confluence Stm Exp true false confluence The expression must be of type boolean; if it evaluates to true, the given statement is executed, otherwise not. The expression must be of type boolean; if it evaluates to true, Statement-1 is executed, otherwise Statement-2 is executed. [syntax] [semantics] [syntax] [semantics]

62. [ 62 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Control Structures (cont’d) while : for : while ( Exp ) Stm for (Exp 1 ; Exp 2 ; Exp 3 ) Stm Equivalent to: The expression must be of type boolean; if it evaluates to false, the given statement is skipped, otherwise it is executed and afterwards the expression is evaluated again. If it is still true, the statement is executed again. This is continued until the expression evaluates to false. { Exp1; while ( Exp2 ) { Stm Exp3; } } Stm Exp true false confluence Exp 1 ; Exp 2 true false Stm confluence Exp 3 ; [syntax] [semantics] [syntax] [semantics]

63. [ 63 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Which diagram corresponds to the "do-while" statement? A. B. C. D. A. C. D. B.

64. [ 64 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Example (Control-Flow Graph) Example Program: Control-Flow Graph: void m(int account, int rate) { account = account + 10; if (account > 100) rate = rate * 2; else rate = rate / 2; out(“account = ” + account + “ rate = ” + rate); } account = account + 10; rate *= 2; rate /= 2; account > 100 true false out(“account = ” + …); confluence

65. [ 65 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Exercise (groups of 2x): Make a CFG for...: public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println("min=" + mi + "," + "max=" + ma); }} /* 1 if-else */ /* 2 for */ /* 3 if-else */ /* 4 if */ if else for if else if

66. [ 66 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Swap with another group and check (correct) their CFG CFG: int mi, ma; System.out.println ("No numbers"); args.length == 0 mi = ma = Integer.parseInt(args[0]); int i=1; i < args.length i++; System.out.println( " min=" + mi + "," + "max=" + ma); int obs = Integer.parseInt(args[i]); obs > ma ma = obs; mi < obs mi = obs; true false true false true false true false 2 3 4 1

67. [ 67 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Group hand-in for next week public List merge(List list1, List list2){ // invariant: we ASSUME list1 and list2 are sorted! List list3 = new ArrayList (); for (int i=0; i < list2.size(); i++) { for (int j=0; j < list1.size(); j++) { if (list2.get(i) > list1.get(j)) { list3.add(list1.get(j)); list1.remove(j); } list3.add(list2.get(i)); } if (list1.size() > 0) { list3.addAll(list1); } return list3; } Make a perfect! CFG (Control-Flow Graph) for the following program (in your groups):

68. [ 68 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Control Structures (cont’d) ” ?: ”; ”conditional expression”: ” && ”; ”lazy conjunction” (aka., ”short-cut  ”): ” || ”; ”lazy disjunction” (aka., ”short-cut  ”): switch : switch ( Exp ) { Swb * } case Exp : Stm * break; default : Stm * break; Swb: Exp 1 ? Exp 2 : Exp 3 Exp 1 && Exp 2 Exp 1 || Exp 2

69. [ 69 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Control Structures (cont’d 2 ) try-catch-finally (exceptions): return / break / continue : ”method invocation”: e.g.; ”recursive method invocation”: e.g.; ”virtual dispatching”: e.g.; try Stm 1 catch ( Exp ) Stm 2 finally Stm 3 f(x) return ;return Exp ; break ;continue ; f(x)

70. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING White-Box Testing Claus Brabrand [ brabrand@itu.dk ] ( First-year Project Course, ITU, Denmark )

71. [ 71 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White-Box Testing (usually) if : TEST condition true and false if-else : TEST condition true and false while : TEST zero, one, more-than-one iterations in loop for : TEST zero, one, more-than-one iterations in loop

72. [ 72 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White-Box Testing ”Coverage Table”: ”Expectancy Table”: Input property account > 90 account <= 90 Data set A B Choice 1 ife true false Data set A B   Input (98,3) (76,4) Expected output ”a=108, r=6” ”a= 86, r=2” Actual output ”a=108, r=6” ”a= 86, r=2” = void m(int account, int rate) { account = account + 10; if (account > 100) // 1 ife rate = rate * 2; else rate = rate / 2; out(“account = ” + account + “ rate = ” + rate); } Testing the world famous method: ”insert-$10-then-double-rate-if-account- exceeds-$100-otherwise-halve-rate”

73. [ 73 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 ”Coverage Table”: Coverage Table Input property No numbers At least one number Exactly one number Exactly two numbers At least three numbers " N > current max" " N  current max" " N  cur max & N > cur min" " N  cur max & N  cur min" Data set A B B C E C D E (3 rd num) E (2 nd num) Choice 1 ife true false 2 for zero-times once more-than-once 3 ife true false 4 if true false NB: you don't (necessarily) have to "pack" into minimal data set

74. [ 74 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 ”Expectancy Table”: Expectancy Table Data set A B C D E      Advice: Avoid expected 0’s (i.e., zeroes) (Default value in many languages.) Advice: Avoid reusing same numbers in tests (Data layout sometimes reuse old memory.) Input  [17] [27,29] [39,37] [49,47,48] Expected output ”no numbers” ”min=17,max=17” ”min=27,max=29” ”min=37,max=39” ”min=47,max=49” Actual output ”no numbers” ”min=17,max=17” ”min=27,max=29” ”min=39,max=39” ”min=49,max=49” =

75. [ 75 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma); }} Debugging ’ D ’ then reveals… /* 1 if-else */ /* 2 for */ /* 3 if-else */ /* 4 if */ if else for if else if Should have been: (mi > obs)

76. [ 76 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Re-Test ! …as debugging often introduces new errors ! Fixed Program: Coverage Table: Expectancy Table:  Recall: no guarantee!

77. [ 77 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Another Example public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1 if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3 if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4 for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6 if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2); } } }

78. [ 78 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Coverage Table Choice Input property Data set 1 ife true No numbers 1 ife false At least one number 2 ife true Exactly one number 2 ife false At least two numbers 3 if true 2 nd number ≥ 1 st number 3 if false 2 nd number < 1 st number 4 for zero-times Exactly two numbers 4 for once Exactly three numbers 4 for more-than-once At least four numbers 5 ife true 3 rd number < current min 5 ife false 3 rd number ≥ current min 6 if true 3 rd ≥ cur min & 3 rd < 2 nd least 6 if false 3 rd ≥ cur min & 3 rd ≥ 2 nd least ABBCCDDEHEFFGABBCCDDEHEFFG

79. [ 79 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Expectancy Table Data set Input Expected output = Actual output A  ”no numbers” ”no numbers” B [17] ”17” ”17” C [27,29] ”27 and 29” ”27 and 0” D [39,37] ”37 and 39” ”37 and 39” E [49,48,47] ”47 and 48” ”47 and 48” F [59,57,58] ”57 and 58” ”57 and 58” G [67,68,69] ”67 and 68” ”67 and 0” H [77,78,79,76] ”76 and 77” ”76 and 77”         Debugging reveals that variable ” mi2 ” erroneously retains initialization (0).

80. [ 80 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Debugging public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1 if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2 if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3 if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4 for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5 if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6 if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2); } } } mi2 = obs; Re-Test: 

81. [ 81 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Which cases are "covered" by input "n=0"? A. if/false, while/zero B. if/false C. if/true D. if/true, while/zero E. if/true, while/zero, while/one, while/many F. none of the above answers are correct void mth(int n) { if (n!=0) { /* if */ while (n>0) { /* while */ n--; }

82. [ 82 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 White-Box Test Exercise Make a systematic white-box test of the following program (iterative factorial): i.e., you need to make: Coverage Table Expectancy Table (w/o "actual output") int factorial(int n) throws BadUserException { if (n<0) throw BadUserException; else { int res = 1; for (int i=1; i<=n; i++) { res = res * i; } return res; }

83. [ 83 ] Claus Brabrand, ITU, Denmark TESTINGFeb 02, 2010 Group hand-in for next week public List merge(List list1, List list2) { // invariant: we ASSUME list1 and list2 are sorted! List list3 = new ArrayList (); for (int i=0; i < list2.size(); i++) { for (int j=0; j < list1.size(); j++) { if (list2.get(i) > list1.get(j)) { list3.add(list1.get(j)); list1.remove(j); } list3.add(list2.get(i)); } if (list1.size() > 0) { list3.addAll(list1); } return list3; } Note: Do not forget to always re-test after debugging! Make a perfect! systematic white-box test of the following program (in your groups):

84. Claus Brabrand, ITU, Denmark Feb 02, 2010TESTING Thanks! Get to know each other in your group – and discuss how you would like to work together