1. Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Black-Box Testing Claus Brabrand [ brabrand@itu.dk ] ( “FÅP”: First-year Project Course, ITU, Denmark )

2. [ 2 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Reminder: Learning & Exam Goals ”Product”: ”Oral Exam”:

3. [ 3 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Outline Introduction: ”Bugs” and ”Black-box testing” Testing the Specification: [”static black-box testing”] Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning” Other techniques: Model-checking & Static-analysis Exercises: Work on your group exercises

4. [ 4 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Definition: ”bug” Main entry: 2 bug Pronunciation: /’bəg/ Function: noun Etymology: origin unknown Date: 1622 1 a: an insect or other creeping or crawling invertebrate (as a spider or centipede) b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious c: any of an order (Hemiptera and especially its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests —called also true bug 2: an unexpected defect, fault, flaw, or imperfection 3 a: a germ or microorganism especially when causing disease b: an unspecified or nonspecific sickness usually presumed due to a bug 4: a sudden enthusiasm 5: enthusiast 6: a prominent person 7: a crazy person 8: a concealed listening device 9: a weight allowance given apprentice jockeys

5. [ 5 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Example of a Bug :-) Hej Claus, A propos test og datoberegninger: Fredag morgen mente min PDA kalender (Microsoft Windows CE) at vi skrev lørdag 1. marts. Jeg stillede selvfølgelig tålmodigt uret tilbage til fredag 29. februar 2008 (dette fandt sted på et møde hos Microsoft i Vedbæk ;-) Lørdag formiddag startede den så med at vise min (ret tomme) kalender for august 2049. Det indbyggede ur stod på 1. januar 1601. Ahem. Peter

6. [ 6 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Terminology (for ’Bugs’) ”Severe conditions”: Fault Failure Defect ”Unintended operation”: Anomaly Incident Variance Feature (intended) ”Generic terms”: Problem Error Bug (  we’ll use this term) Typically imply blame; as in: - ”it was his fault” Famous quote: - ”It’s not a bug, it’s a feature”

7. [ 7 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING ”The Harvard Mark II Bug” Photo of first actual ”bug”: “The first documented computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested” Harvard University, Sep. 9, 1947

8. [ 8 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Definition: ”Bug” A ”bug” is a relation between: Specification & Implementation Whether or not specification is: Explicit or Implicit Written or Oral Formal or Informal (e.g., ”product spec” vs. ”back-of-envellope”) implementation (aka., ’program’) specification (aka., ’spec’) ~ bug

9. [ 9 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Specification A spec states things an impl…: Should do! Shouldn’t do! Unspecified? (’unclear’, ’unmentioned’, or ’left open’) Should do! Shouldn’t do! Unspecified? S Specs are often intentionally under-specified. It’s often better to not ”prematurely commit” to a particular solution (by specifying exactly how a task should be done) – and instead just state which overall tasks the system should do.

10. [ 10 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Formal Definition: ”Bug” A (software) bug occurs when: 1. Impl doesn’t do sth spec says it should: 2. Impl does sth spec says it shouldn’t: 3. Impl does sth spec doesn’t mention: 4. Spec doesn’t mention sth, but should: 5. Impl is hard to understand/use by user(s): ~ ~ ?! (or does it incorrectly) I S I S S I ?! I S [cf. ”Software Testing”, R.Patton, p.15] ! ~ ideal

11. [ 11 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Consequences… Consequently, …: ”Additional functionality”? e.g., a calculator which also does square root (but wasn’t supposed to) ”Easter egg”? e.g., hit [Alt+Shift+2] in Solitaire to win game …is a bug !

12. [ 12 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Question Question: A) R.Patton interprets as (”spec := impl”):  trivially no bugs in impl ! (according to [own] definition #1,#2,#3) B) More sensible to interpret as ”no spec”:  entire impl is essentially one big bug ! (…according to definition #3+#4 [p.15]) ”Because the software is already complete, you have the perfect specification – the product itself” [R.Patton, ”Software Testing”, p.31] ”an impl w/o a spec” ? bug ~

13. [ 13 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Often due to:  complexity  tight schedule  under-specification  under-documentation Number one Cause of Bugs… [cf. ”Software Testing”, R.Patton, p.17] (NB: no percentages given) spec …is the specification: design impl other

14. [ 14 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Cost of (Fixing) Bugs Cost of bugs increases exponentially (over time): log specdesigncodetestrelease $1 $100 $10,000 $1,000,000 $10,000,000 [cf. ”Software Testing”, R.Patton, p.18]

15. [ 15 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Tester’s Goal The goal of a software tester is: Testers (therefore) often aren’t the most popular members of a project team - 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed” [R.Patton, “Software Testing”, p. 19] [cf. ”Software Testing”, R.Patton, p.19]

16. [ 16 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING ”The Pesticide Paradox” ”The pesticide paradox”: ”The more you test a software, the more immune it becomes to your tests” B.Beizer, “Software Testing Techniques”, 1990

17. [ 17 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Bugs Bugs often occur in groups If you find one, chances are there are more nearby Not all bugs will be fixed Too costly? Too risky? Not ”cost-effective”? Bug unknown :-)

18. [ 18 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Programming vs. Testing Constructive thinking: ”Test-to-pass” Often not a good idea to ”test” your own code :-( Destructive thinking: ”Test-to-fail” (aka., ”error-forcing”) Often better to test/break someone else’s code :-) Advice: ”test-to-pass” before ”test-to-fail” (i.e., test ”normal usage” first)

19. [ 19 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Black-Box Testing The goal of black-box testing is: Make sure impl solves problem it is supposed to: i.e., Point of departure: spec, not impl not a particular program which ”claims” to solve problem testing w/o insights of code impl ~ spec Static (test spec): Dynamic (test impl): output input bug ~

20. [ 20 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Outline Introduction: ”Bugs” and ”Black-box testing” Testing the Specification: [”static black-box testing”] Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning” Other techniques: Model-checking & Static-analysis Exercises: Work with your group exercises

21. [ 21 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Spec ”Warning Words” (I/III) Unconditionals (universally): ’Always’, ’for every’, ’for each’, ’for all’, … Try to violate (i.e., find exemptions from rule)! Unconditionals (never): ’None’, ’never’, ’not under any circumstances’, … Try to violate (i.e., find exemptions from rule)! Tautologies: ’Trivially’, ’certainly’, ’clearly’, ’obviously’, ’evidently’, … Check assumptions (that nothing’s swept under the rug)! [cf. ”Software Testing”, R.Patton, p.61]

22. [ 22 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Spec ”Warning Words” (II/III) Unspecified conditionals: ’Some(-times)’, ’often’, ’usually’, ’ordinarilly’, ’mostly’, … Unclear spec (under which circumstances)? Continuations: ’Etcetera’, ’and so forth’, ’and so on’, … Check that spec is comprehensively unambiguous? Examples: ’E.g.’, ’for example’, ’such as’, … Is example representative (what about other examples)? [cf. ”Software Testing”, R.Patton, p.61]

23. [ 23 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Spec ”Warning Words” (III/III) Positive adjectives: ’Good’, ’fast’, ’efficient’, ’small’, ’reliable’, ’stable’, … Subjective (needs objectification if to be used for testing)! Alegedly completed: ’Handled’, ’processed’, ’taken care of’, ’eliminated’, … Is something hidden? Incompleted: ’Skipped’, ’unnecessary’, ’superfluous’, ’rejected’, … Is something forgotten? [cf. ”Software Testing”, R.Patton, p.61]

24. [ 24 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Spec ”Warning Words” (if-then) Finally, watch out for: ”If … Then” (with missing ”Else”): Check what happens in the ”Else-case” [cf. ”Software Testing”, R.Patton, p.61]  IF …  THEN …  ELSE … ?!

25. [ 25 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Question: which is ”best” (…and why)? A) white-box testing ; black-box testing ; (i.e., white-box testing first) B) black-box testing ; white-box testing ; (i.e., black-box testing first) Answer: ’B’ Settle overall impl ~ spec problems first Before zooming in on the impl [as in white-box testing] Test Sequencing …or… ~

26. [ 26 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Outline Introduction: ”Bugs” and ”Black-box testing” Testing the Specification: [”static black-box testing”] Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning” Other techniques: Model-checking & Static-analysis Exercises: Work on your group exercises

27. [ 27 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Test ”Boundaries” Programs are vulnerable ”around the edges”: e.g. testing legal inputs (time, in hours): e.g. testing legal inputs (dates, in April): Property Minimum-1 Minimum Middle repr. Maximum Maximum+1 Input 0 11 (e.g.) 23 24 Expected output invalid valid invalid Actual output Property Minimum-1 Minimum Middle repr. Maximum Maximum+1 Input 00/4 01/4 12/4 (e.g.) 30/4 31/4 Expected output invalid valid invalid Actual output

28. [ 28 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Test ”Powers-of-Two” Programs vulnerable ”around powers-of-two”: e.g. years of age (assume held in a byte): e.g. #game-spectators (assume held in a 16-bit word): Property Minimum-1 Minimum Middle repr. Maximum Maximum+1 Input 0 27 (e.g.) 255 256 Expected output invalid valid invalid Actual output Property Minimum-1 Minimum Middle repr. Maximum Maximum+1 Input 0 12345 (e.g.) 65535 65536 Expected output invalid valid invalid Actual output

29. [ 29 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Test ”Empty Input” Default / empty / blank / null / zero / none /  : e.g., ’any program’: Property No input Input  Expected output Error message Actual output

30. [ 30 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Test ”Invalid Input” Invalid / illegal / wrong / garbage / bogus data: e.g., calculator: Property Invalid input Bogus data!!! Input +*31 #$+~´?=! Expected output Error message Actual output

31. Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Equivalence Partitioning Partitions and Equivalence Relations

32. [ 32 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Testing: Infinite process Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”) There are infinitely many possible inputs: (hence, testing will take an infinite amount of time) sum(x,y) (0,0) 0 sum(x,y) (1,0) 1 sum(x,y) (0,1) 1 sum(x,y) (-1,0) sum(x,y) (0,-1) sum(x,y) (1,1) 2 sum(x,y) (-1,-1) -2 …

33. Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Crash course on Relations Relations Equivalence Relations

34. [ 34 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Relations Example 1 : “even” relation: Written as: as a short-hand for: … and as: as a short-hand for: Example 2 : “equals” relation: Written as: as a short-hand for: … and as: as a short-hand for: Example 3 : “dist-btwn” relation: Written as: as short-hand for: | _ even  Z | _ even 4 | _ even 5 4  | _ even 5  | _ even 2  3 (2,3)  ‘=’ ‘=’  Z  Z (2,2)  ‘=’ 2 = 2 ‘  ’  CITY  Z  CITY Aarhus  Copenhagen 310 (Aarhus, 310, Copenhagen)  ‘  ’

35. [ 35 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Equivalence Relation Let ‘~’ be a binary relation over set A: ‘~’  A  A ~ is an equivalence relation iff: Reflexive: Symmetric: Transitive:  x  A: x ~ x  x,y  A: x ~ y  y ~ x  x,y,z  A: x ~ y  y ~ z  x ~ z  

36. [ 36 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Exercise: Eq.Rel. Which relations are equivalence relations?: …and which are not (and why not)?: a) The "less-than-or-equal-to" relation: '  ' { (n,m) | n,m  Z, n  m } b) The ”almost-total-relation-on-integers”, (relating all numbers except 42, but relating 42 with 42): { (n,m) | n,m  ( Z \ {42} ) }  { (42,42) } c) The "is-congruent-modulo-three" relation: { (n,m) | n,m  Z, (n % 3) = (m % 3) } d) The "have-talked-together" relation: { (p,q) | n,m  PEOPLE, p and q have talked together } e) The "is-in-the-same-group-as" relation: { (p,q) | n,m  PEOPLE, p and q are in same FÅP group }

37. [ 37 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Eq. Rel.  Partition Equivalence relation ’~’ = { (A,A), (B,B), (A,B), (B,A), (P,P), (X,X), (Y,Y), (Z,Z), (X,Y), (Y,X), (X,Z), (Z,X), (Y,Z), (Z,Y) } E.g.: A ~ B, P ~ P, X ~ X, X ~ Z A ~ P, B ~ Y, P ~ Z Partition ’~’: A B P X Y Z ”Canonical representatives”: [A] = [B] = { A, B } [P] = { P } [X] = [Y] = [Z] = { X, Y, Z } Capture the same information; i.e., notion of ”equivalence”

38. [ 38 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Testing: Infinite process Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”) There are infinitely many possible inputs: (hence, testing will take an infinite amount of time) sum(x,y) (0,0) 0 sum(x,y) (1,0) 1 sum(x,y) (0,1) 1 sum(x,y) (-1,0) sum(x,y) (0,-1) sum(x,y) (1,1) 2 sum(x,y) (-1,-1) -2 …

39. [ 39 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Equivalence Partitioning Partition input: Finitary partition: If finite # categories (aka. ”equivalence classes”) Here 3: { ”zero”, ”pos”, ”neg” } We can now test all equivalence classes Using representative elements from each category negpos 0 zero 1 2 3 …… -3 -2 -1

40. [ 40 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Test Sum (cont’d) We can now test all equivalence classes Using representative input from each category Sum (testing all equivalence classes): Property Pos, Pos Neg, Pos Zero,Pos Pos, Neg Neg, Neg Zero,Neg Pos, Zero Neg, Zero Zero,Zero Input (1,2) (-3,4) (0,5) (6,-7) (-8,-9) (0,-10) (11,0) (-12,0) (0,0) Expected output 3 1 5 -17 -10 11 -12 0 Actual output

41. [ 41 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Frequent Partitions for Testing Numbers: Positive, Negative, Zero Zero, One, Two, Many (aka. ”Greenlandic Numbers”) Lists: Length-0, Length-1, Length-2, Length-3+ Ascending-elements, Descending-elements, AscDesc-elements … Advice: - consider how problem might be solved - partition into qualitatively different categories such that: (likely: ”same category  same error”)

42. [ 42 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Outline Introduction: ”Bugs” and ”Black-box testing” Testing the Specification: [”static black-box testing”] Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning” Other techniques: Model-checking & Static-analysis Exercises: Work on your group exercises

43. Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Model-checking Modelling Verification Model-checking

44. [ 44 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Program world Model world Concrete Abstract P M Program vs. Model World abstraction concretization (Java) (modeling lang)

45. [ 45 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Methodology: Model-based Design Design abstract model Decompose model Reason/Test/Verify model individual parts and whole Recompose insights make model safe Impl. concrete program concretize REAL PROBLEM SAFE MODEL SAFE PROGRAM abstract ? ? ? ? MODEL test reason verify

46. [ 46 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Program world Model world Concrete Abstract P M 1. P  ?3. M  ? 4. verify!! 5. M  ! 7. P  ! 2. abstract 6. concretize Model-checking (Java) (modeling lang) If models are “finite”, we can have a computer test all possibilities. “Never send a human to do a machine’s job” -- A.Smith (’99)

47. Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Static Analysis Undecidability

48. [ 48 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Undecidability Most interesting properties are undecidable: e.g., can program P have a type error (when run): Compilers use approximations (computed via ”static analyses”): correct incorrect ? correct incorrect safe (over-)approximation no program can ”decide” this line automatically (in all cases)

49. [ 49 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Undecidability (self-referentiality) Consider "The Book-of-all-Books": This book contains the titles of all books that do not have a self-reference (i.e. don't contain their title inside) Finitely many books; i.e.: We can sit down & figure out whether to include or not... Q: What about "The Book-of-all-Books" ; Should it be included or not? "Self-referential paradox" (many guises): e.g. "This sentence is false" "The Bible" "War and P e ac e" "Programm i n g La nguages, An Interp.- B a s e d Approach"... The Book-of-all-Books

50. [ 50 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Termination Undecidable! Assume termination is decidable (in Java); i.e.  some program, halts: PROG  BOOL Q: Does P 0 loop or terminate...?:) Hence: "Termination is undecidable"...for Java, C, C++, Pascal, -Calculus, Haskell,... bool halts(Program p) {... } Program p0 = read_program("P 0.java"); if (halts(p0) == “halts”) loop(); else halt(); -- P 0.java -- bool halts(Program p) {... }

51. [ 51 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Example: Type Checking Will this program have type error (when run)? Type error  evaluates to true void f() { var b; if ( ) { b = 42; } else { b = true; } /* some code */ if (b)...; } Undecidable (in all cases) i.e., what evaluates to (when run)

52. [ 52 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Example: Type Checking (cont’d) Hence, languages use static requirements: All variables must be declared And have constant type (throughout the program) void f() { bool b; /* some code */ if ( ) { b = 42; } else { b = true; } /* some more code */ } Static compiler error: Regardless of what evaluates to when run

53. [ 53 ] Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING Outline Introduction: ”Bugs” and ”Black-box testing” Testing the Specification: [”static black-box testing”] Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning” Other techniques: Model-checking & Static-analysis Exercises: Work on your group exercises