DAIMI(c) Henrik Bærbak Christensen1 BlackBox Testing based on Specification

DAIMI(c) Henrik Bærbak Christensen2 Classes of tests Black-box testing (functional/spec testing) No knowledge of realization of specification; only specification is available... Unit Under Test input output Specification

DAIMI(c) Henrik Bærbak Christensen3 Classes of tests White-box testing (glass-box / structural) Complete knowledge of realization. input output

DAIMI(c) Henrik Bærbak Christensen4 The problem of testing How do we know that our program is free of defects? Exhaustive input testing: All possible input! Does it scale? Is it feasible ? UUT input

DAIMI(c) Henrik Bærbak Christensen5 Principle When exhaustive testing is out of question the question is how to get good testing at the smallest possible cost? Principle 2: Good test case has high probability of detecting a defect. Systematic testing is about finding good test cases. BB is when you analyze the specification in order to identify this set.

DAIMI(c) Henrik Bærbak Christensen6 Equivalence Class Partitioning Insight: –Many input are processed by the same algorithm part thus if input A is processed by the same algorithm as input B then adding a test case for B does not increase our belief in the algorithm’s reliability much. –Example: System.abs(x) System.abs(37) == 37 pass Will x = 42 have a high probability of finding a defect? Will x = ? Will x = -37

DAIMI(c) Henrik Bærbak Christensen7 Equivalence Class Partitioning Thus we divide the input space into partitions named Equivalence Classes (ECs) with the representation property Representation: –if one element x in a EC expose a defect then all other elements in EC will expose the same defect If System.abs(37) == 37 fails I am highly confident that so will System.abs(39) == 39...

DAIMI(c) Henrik Bærbak Christensen8 ECs ECs allow us to reduce the number of test cases by a very high factor –System.abs: from 2^32 to 2. –Below: From N to 4. ECs are sets (Da: mængder) Unit Under Test EC1 EC2 EC4 EC3

DAIMI(c) Henrik Bærbak Christensen9 The process The process is –Find the ECs from the specification –For each EC pick one element (the representative) and define a test case from it. –Iterate if you get nervous that you have missed something -> repartition the ECs System.abs example –The spec talks about negative and positive x EC1: x > 0; EC2: x < 0 –use x = 37 as representative for EC1; x=-42 for EC2

DAIMI(c) Henrik Bærbak Christensen10 The devil is in the detail ! The often hard problem is finding the right ECs. I have tried for years to provide a solid ground by a method proposed by Myers and throwing some set theory after it. My students generally have problems –the process and math makes them partially blind to the goal: finding good test cases! Thus – we will keep it at an intuitive level –maybe this will work better

DAIMI(c) Henrik Bærbak Christensen11 Partitioning Heuristics 1) If you have a range of values specification –make 3 ECs: [1] in rangevalid [2] above rangeinvalid [3] below rangeinvalid dayOfWeek(int year, int month, int date) // year in [ ], month in [1-12] –3 ECs for year; 3 ECs for month

DAIMI(c) Henrik Bærbak Christensen12 Partitioning Heuristics 2) If you have a set, S, of values specification –make |S|+1 ECs [1]..[|S|] one for each member in Svalid [|S|+1 for a value outside Sinvalid A vending machine accepts 1,2,5,10,20 kr coins –6 ECs{1}, {2}, {5}, {10}, {20}, {17}

DAIMI(c) Henrik Bærbak Christensen13 Partitioning Heuristics 3) If you have a boolean condition specification –make 2 ECs the first character must be non-digit –2 ECs: {“a1”}, {“1a”} the object reference must not be null –you get it, right?

DAIMI(c) Henrik Bærbak Christensen14 Partitioning Heuristics 4) If you question the representation property of any EC, then repartition! –Split EC into smaller ECs

DAIMI(c) Henrik Bærbak Christensen15 Valid and Invalid ECs Burnstein (and Myers whom described the technique back in the 70’s) classify ECs as either valid or invalid. The words can be misinterpreted because it is not always a matter of the input being valid or not. I suggest that you consider it to define two classes of method processing: –end-to-end processing ECs = “valid” –bail-out processing ECs = “invalid” This is arguably an algorithmic viewpoint (whitebox)

DAIMI(c) Henrik Bærbak Christensen16 ECs classes Consider an archetypical algorithm/method: function algorithm(p1,p2,p3) { if ( ! valid(p1) ) { return; } if ( ! valid(p2) ) { return; } if ( ! valid(p3) ) { return; } A = calculateA(p1,p2,p3) if ( abnormal-condition(A) ) { return; } B = calculateB(p1,p2,p3) if ( abnormal-condition(B) ) { return; } C = calculateC(p1,p2,p3) return C; } A normal operation/normal use case will make the function process end-to-end –p1,p2,p3 are all used in processing the result An error condition/abnormal processing will make the function bail-out –only pN is used to process some partial result end-to-end bail-out

DAIMI(c) Henrik Bærbak Christensen17 Examples function algorithm(year) { if (year < 1900 ) { throw new IllegalArgumentException(); } if (year > 3000 ) { throw new IllegalArgumentException(); } calculate using a valid year Here elements from the bail-out/invalid ECs will terminate the function before it reaches the end.

DAIMI(c) Henrik Bærbak Christensen18 Making TestCases Now the process to make test cases is: 1) Make a test case that combines as many end- to-end/valid ECs as possible –repeat until all valid ECs are covered... 2) Make a test case that picks an element from o ne bail-out/invalid EC at a time – all other elements must be from e2e ECs –repeat for every invalid EC

DAIMI(c) Henrik Bærbak Christensen19 Example boolean CPRTest(String cpr) –“ ” form C1) Must be 11 characters C2) Pos 6 must be ‘-’ C3) Two first characters must be in range C4) Two next characters must be in range 01-12

DAIMI(c) Henrik Bærbak Christensen20 ECs C1) [1] 11 chars [2] not 11 char C2) [3] p(6)==‘-’ [4] p(6) != ‘-’ C4) [5] 31 C4) [8] 12 TestCases – let us start with the bail-outs –there are 6 of them. We must take one at the time –TC1: “ ” Important note!!! It is only [2] that is in play, no other invalid EC is in play !!! –TC2: “111106x1232” Again, do not combine invalid/bail-out ECs –Stay tuned for the reason

DAIMI(c) Henrik Bærbak Christensen21 Test Case generation Next we combine as many e2e ECs as possible. This is easy! We only need one, namely a completely valid CPR number.

DAIMI(c) Henrik Bærbak Christensen22 Why the difference? Bail-out ECs masks each other and defects keep lurking in there Example: –TC: “xxxx” This will return false, it is not a CPR. But it only demonstrates that one condition is programmed correctly (maybe length check) while all others may be obscurely wrong...

DAIMI(c) Henrik Bærbak Christensen23 function cpr(String s) { if ( s.length() != 11 ) { throw new IllegalArgumentException(); } if ( s.charAt(5) != ‘-’ ) { throw new IllegalArgumentException(); } etc. This is called masking. The first test masks the defect in the next. Therefore only make one element invalid at a time...

DAIMI(c) Henrik Bærbak Christensen24 Valid ECs On the other hand we can often combine as many valid ECs as possible because the valid EC elements are not treated individually but combined algorithmically. Thus there is a high probability that a wrong computation will show up. Ex: day of week calculation uses both year, month, date and a defect will (probably) show up in any combination we make... –leap years will require repartitioning the valid ECs

DAIMI(c) Henrik Bærbak Christensen25 What does valid/invalid mean? Students complain about “invalid”. What does it mean? And I cannot give the answer. Example: Backgammon move validation –validity = v(board-state, move, player, die) It is completely valid input parameters to test a move that the v method calculate as invalid. –thus there are no “invalid” ECs However – the algorithm is full of bail-outs! –all individual “invalid move” conditions will be tested one by one by the algorithm – resulting in bail-outs Result: There must be a test-case for each potential rule that judge a move as invalid: wrong distance, checker in bar, etc.

DAIMI(c) Henrik Bærbak Christensen26 Take care of the invalid ones!!! I have seen many invalid ECs that were irrelevant!!! You should not consider invalid ECs with impossible elements !!! –You cannot move a negative number of checkers when you have a direct manipulation graphical user interface!!!

DAIMI(c) Henrik Bærbak Christensen27 Take care of the invalid ones!!! Thus: Test cases must respect the preconditions of the UUT. If the method states that it will assumes parameters within range you cannot define test cases where they are outside. If not you end up introducing parameter checking code in every method in every class. Keep your parameter checking at the user interface and let the domain code run fast!

DAIMI(c) Henrik Bærbak Christensen28 Boundary Value Analysis

DAIMI(c) Henrik Bærbak Christensen29 Boundary value analysis Experience shows that test cases focusing on boundary conditions have high payoff. Some that spring into my mind is –iteration over a C array at its max size –”off by one” errors in comparisons < used but should have been a <= –null as value for a reference/pointer

DAIMI(c) Henrik Bærbak Christensen30 Boundaries Where are the boundaries? Between the ECs! Example: EC: [1] 3000 As the processing of 1899 is radically different from that of 1900, testing with the boundary value 1900 seems like a very good choice.

DAIMI(c) Henrik Bærbak Christensen31 Other techniques

DAIMI(c) Henrik Bærbak Christensen32 State-Transition Testing If you have state machines, make test cases that ensures that all state changes are exercised...

DAIMI(c) Henrik Bærbak Christensen33 Error-Guessing Use your intuition and experience Test-Driven Development is based upon this “happy-go-lucky” idea.

DAIMI(c) Henrik Bærbak Christensen34 Summary BlackBox: Make test cases from specs. EC partitioning: –Identify large sets of input that can be assumed to be processed identically, and make a test case with only a single candidate element from this EC –Make test cases by combining ECs only one “invalid” at a time as many “valid” as possible Boundary value analysis –The defects are at the EC boundaries...