Software Testing Sudipto Ghosh CS 406 Fall 99 November 9, 1999

11/09/99CS 406 Testing2 Learning objectives Functional testing Equivalence class partitioning Boundary value analysis Structural testing Control flow-based

11/09/99CS 406 Testing3 Testing for correctness Identify the input domain of P. Execute P against each element of the input domain. For each execution of P, check if P generates the correct output as per its specification S.

11/09/99CS 406 Testing4 Input domain The set of all valid inputs that a program P can expect is the input domain of P. The size of an input domain is the number of elements in it. An input domain could be finite or infinite. Finite input domains may be large!

11/09/99CS 406 Testing5 Identifying input domains For the sort program: N: size of the sequence, N < 3 Sequence of N integers, such that for any integer K, 0  K  (e –1), e = 3 K: each element in the sequence Some sequences are: [ ] : An empty sequence (N = 0) [0] : A sequence of size 1 (N = 1) [2 1] : A sequence of size 2 (N = 2)

11/09/99CS 406 Testing6 Size of input domain Suppose that 0  N  10 6 The size of the input domain is the number of all sequences of size 0, 1, 2… The size is computed as: To test for correctness, P needs to be tested on all inputs. How many years would that take?

11/09/99CS 406 Testing7 Exhaustive testing Previous example P is executed on all elements in the input domain. For most programs, exhaustive testing is not feasible What is the alternative?

11/09/99CS 406 Testing8 Partition testing Input domain is partitioned into a finite number of sub-domains. P is then executed on a few elements in each sub-domain. Four sub-domains for our example: [], [2], [2 0], [2 1 0] Reduced from 85 to 4 ! N=0 1 N=1 2 N=3 4 N=2 3

11/09/99CS 406 Testing9 Confidence in your program Confidence is a measure of one’s belief in the correctness of the program Correctness is not measured in binary terms: a correct or incorrect program Measured as a probability of correct operation of a program when used in various scenarios Reliability Test completeness: Extent to which a program has been tested and errors have been found and removed

11/09/99CS 406 Testing10 Types of module testing “Informal testing” performed by programmer while developing module Methodical testing performed by SQA group Nonexecution-based testing Execution-based testing Haphazard data as input Systematically constructed test cases Testing to specifications Testing to code

11/09/99CS 406 Testing11 Testing to specifications Black-box testing Data-driven testing Input/output-driven testing Functional testing Code is ignored, i.e. the internal structure of the code is not important The specification document is the only information used for creating test cases

11/09/99CS 406 Testing12 Testing to code White-box testing Glass-box testing Logic-driven testing Path-oriented testing Specifications are not used to generate the test cases Test “to code,” i.e. the internal structure of the code is important

11/09/99CS 406 Testing13 What is functional testing? Functional testing tests how well a program meets the functionality requirements. When test inputs are generated using program specifications, we say that we are doing functional testing.

11/09/99CS 406 Testing14 Feasibility of functional testing Problem with exhaustive testing Very large (possibly infinite) number of test cases. Need to devise: Small, manageable set of test cases. Maximize the chances of detecting faults. Minimize the chances of wasting test cases by having more than one test case detecting the same fault.

11/09/99CS 406 Testing15 Equivalence class partitioning Divide the domain of all inputs into a set of equivalence classes. If any test in an equivalence class succeeds, then every test in that class will succeed. How would you get ideal equivalence classes? Difficult without looking at the internal structure Difficult even with the internal structure

11/09/99CS 406 Testing16 Equivalence class partitioning Set of all inputs Specifications Put those inputs together for which the behavior pattern of the module is specified to be different into similar groups Equivalence classes

11/09/99CS 406 Testing17 Rationale behind equivalence class partitioning We assume that if the specifications require exactly the same behavior from each element in a class of values, then the program is likely to be constructed so that it either succeeds or fails for each of the values in that class.

11/09/99CS 406 Testing18 Guidelines for partitioning For robust software, we must test for incorrect inputs too. For each equivalence class of valid inputs, we have equivalence classes of invalid inputs. Input condition specifies a range a  X  b. a  X  b (Valid case) X b (invalid cases) Input specifies a value create one for the valid value create two for incorrect (one above, one below)

11/09/99CS 406 Testing19 Guidelines for partitioning Input specifies a value (contd.) For boolean, only one value Input condition specifies a member of a set Create one for the valid value Create one for invalid value (not in the set) Example: Factorial (n), where n  0 Valid classes {0}, {x | x  1} Invalid class {x | x < 0}

11/09/99CS 406 Testing20 Non-overlapping partitions In the previous example, the equivalence classes were non-overlapping, i.e., the sub-domains were disjoint. It is sufficient to pick one test from each equivalence class to test the program. An equivalence class is considered covered if at least one test has been selected from it. Our goal is to cover all equivalence classes.

11/09/99CS 406 Testing21 Overlapping partitions Suppose a program P takes three integers, X, Y and Z. It is known that: X < Y Z > Y X < Y Z > Y X < Y X  Y Z  Y

11/09/99CS 406 Testing22 Overlapping partition:test selection Select 4 test sets as X=4, Y=7, Z=1 (satisfies X < Y) X=4, Y=2, Z=1 (satisfies X  Y) X=1, Y=7, Z=9 (satisfies Z > Y) X=1, Y=7, Z=2 (satisfies Z  Y) Can also reduce the number of test cases to 2 X=4, Y=7, Z=1 (satisfies X < Y and Z  Y) X=4, Y=2, Z=3 (satisfies X  Y and Z > Y)

11/09/99CS 406 Testing23 Boundary Value Analysis (BVA) Observation: Programs that work correctly for a set of values in an equivalence class, fail on some special values. These values lie on the boundary of the equivalence class. Choose an input from a test case from an equivalence class such that the input lies at the edge of the equivalence class. These test cases are “extreme cases”

11/09/99CS 406 Testing24 BVA examples Suppose the range is 0.0  X  (valid input) 1.0 (valid input) -0.1 (invalid input) 1.1 (invalid input) For a list first and last element of the list A program takes a string S and integer X as input such that a  X  b, and length(S)  100. Derive tests.

11/09/99CS 406 Testing25 BVA analysis A program takes two integers X and Y, such that a  X  b, c  Y  d. How many test cases do we get? ab c d

11/09/99CS 406 Testing26 Output variables Equivalence class partitioning can be applied to output variables BVA also can be applied to output data

11/09/99CS 406 Testing27 Testing functions Identify each function implemented in a module Devise test data to test each function separately Module may contain a hierarchy of lower level functions connected by program control structures Perform functional testing recursively Higher level: black-box Lower level: glass-box

11/09/99CS 406 Testing28 Problems with previous approach Usually higher level functions are not as structured Lower level functions are intertwined Functionality does not coincide with module boundaries Distinction between module testing and integration testing is blurred Testing one module is not possible without simultaneously testing other modules whose functionality is used

11/09/99CS 406 Testing29 Structural testing Intent is to exercise the different programming structures and data structures used in the program Intent is not to exercise all the different input and output conditions though this may be a by-product Achieve test cases that force the desired “coverage” of different structures Criteria are formal and precise

11/09/99CS 406 Testing30 Control-flow based criteria Statement coverage Run a series of test cases and ensure that every statement is executed at least once Simplest form of glass box testing What is the weakness? Consider the example: int abs (int x) { if (x >= 0) x = 0 - x; return x; } Error Test inputs: x = 5: What is coverage? x = 0: What is coverage?

11/09/99CS 406 Testing31 Statement coverage Not very strong, may leave errors undetected. Examples: if statement without else part conjunctions of predicates In all these cases, all branches were probably not exercised. Can you think of a better criterion based on the above observation?

11/09/99CS 406 Testing32 Branch coverage Require that every decision is evaluated to true and false values at least once during testing Branch coverage implies statement coverage Each statement is part of some branch

11/09/99CS 406 Testing33 Control flow graph of a program Let G be the graph of a program P. Node: Represents a block of statements that are always executed together Edge (i, j) from node i to node j: Represents a possible transfer of control after executing the last statement of the block represented by node i to the first statement in the block represented by node j.

11/09/99CS 406 Testing34 Control flow graph of a program Start node: Node corresponding to a block whose first statement is the start statement of P. Exit node: Node corresponding to a block whose last statement is an exit statement of P. Path: Finite sequence of nodes (n 1, n 2, …, n k ), k>1 such that there is an edge (n i, n i+1 ) for all nodes n i in the sequence, except the last node n k.

11/09/99CS 406 Testing35 Control flow graph of a program Complete Path: Path whose first node is a start node and the last node is an exit node. All-nodes criterion (statement coverage) All-edges criterion (branch coverage)

11/09/99CS 406 Testing36 An example (x y ) 1.scanf(x, y); if(y < 0) 2. pow = 0 – y; 3.else pow = y; 4.z = 1.0; 5.while(pow != 0) 6. { z = z * x; pow = pow – 1; } 7.if ( y < 0 ) 8. z = 1.0/z; 9.printf(z);

11/09/99CS 406 Testing37 Control Flow Graph of example

11/09/99CS 406 Testing38 Problems with branch coverage What if a decision has many conditions (using and, or) Decision may evaluate to true or false without actually exercising all the conditions int check (int x) { if ((x >= 5) && (x <= 200)) return TRUE; return FALSE; } Error (should be 100) Test inputs: x = 5: x = -5:

11/09/99CS 406 Testing39 Solution? Require all individual conditions to evaluate to true and false Problem: Even if individual conditions evaluate to true and false, the decision may not get both true and false values Solution: Require both decision / condition coverage!! Still there is a problem.

11/09/99CS 406 Testing40 Path testing Some errors are related to some combinations of branches. Presence revealed by an execution of a path that includes those branches. Solution: Require all possible paths in the CFG to be executed during testing. Path-coverage criterion, or all-paths criterion Path coverage implies branch coverage