Sudipto Ghosh CS 406 Fall 99 November 16, 1999 Software Testing Sudipto Ghosh CS 406 Fall 99 November 16, 1999

Learning Objectives Data flow-based testing Test adequacy assessment 11/16/99 CS 406 Testing

Problems with path testing With each predicate, there is a combinatorial explosion in the number of paths. Potentially infinite number of paths because of loops. Not all paths a feasible, i.e., there may not be inputs for which the path is executed. Suppose that somehow, we satisfied this criterion. Would we find all the faults? 11/16/99 CS 406 Testing

More problems A path is tested only if it is present!! Consider the example: double logBase19 (double x) { return ln(x)/ln(19); } Missing condition is: if(x > 0) 11/16/99 CS 406 Testing

More problems It is possible to test every path without detecting the fault in the product. Function to test the equality of 3 integers: (Fallacious) assumption: If the average of the 3 numbers is equal to the first, then they are equal. boolean areEqual(int x,int y,int z){ if ((x+y+z)/3 == x) return TRUE; else return FALSE; } Exercise: Think of test cases. 11/16/99 CS 406 Testing

Reducing the number of paths to be covered Find something between branch and path coverage. Select a set of paths that ensure branch coverage and try some other paths that help reveal errors. Consider two paths same if they differ only in their sub-paths that are caused due to loops. Restrict test cases to linear code sequences Identify set of points L, from which control flow may jump; includes entry, exit, branch statements Paths begin and end at elements in L. 11/16/99 CS 406 Testing

Data Flow-Based Testing Select paths to be executed based on data flow analysis Information about where variables are defined variables are used Idea is to make sure that the definitions of variables and their subsequent use is tested 11/16/99 CS 406 Testing

Variables in statements Variables occur in statements as Definition – def Variables on the left side of a statement ( c ) c := 10; Variable is used in a computation – c-use Variables on the right-hand side of a statement ( c ) temp := c + 5; Variables used in a write statement Variable is used in a predicate – p-use if( c!= MAXLENGTH ) { 11/16/99 CS 406 Testing

An example (xy) scanf(x, y); if(y < 0) pow = 0 – y; else pow = y; z = 1.0; while(pow != 0) { z = z * x; pow = pow – 1; } if ( y < 0 ) z = 1.0/z; printf(z); 11/16/99 CS 406 Testing

Def/use graph of example 1 def = {x, y}: c-use = {} y y def = {pow}: c-use = {y} 3 2 def = {pow}: c-use = {y} 4 def = {z}: c-use = {} 5 def = {}: c-use = {} pow pow def = {z, pow}: c-use = {x, z, pow} 6 7 def = {}: c-use = {} y y def = {z}: c-use = {z} 9 def = {}: c-use = {z} 8 11/16/99 CS 406 Testing

More on def/use graphs With each node, we associate the c-uses of a variable With each edge, we associate the p-uses of a variable 11/16/99 CS 406 Testing

Def-clear paths Def-clear path with respect to a variable x: Path from node i to node j, such that there is no def of x in the nodes in the path from i to j. Nodes i and j may have a def. Find out examples from the previous graph. Path from node i to edge (j, k), such that there is no def of x in the nodes in the path. 11/16/99 CS 406 Testing

Global c-use A c-use of a variable is considered global if: there is not def of x within the block preceding the c-use Example? 11/16/99 CS 406 Testing

Global def A def is global if it can be used outside the block in which it is defined. A def of a variable x, in a node i, is a global def, if: it is the last def of x in the block (node) i a def-clear path exists from i to another node which has a global c-use of x Example? 11/16/99 CS 406 Testing

Set def(i) Set of variables for which there is a global def in the node i. Examples: 11/16/99 CS 406 Testing

Set c-use(i) Set of variables for which there is a global def in the node i. Examples: 11/16/99 CS 406 Testing

Set p-use(i, j) For an edge (i, j), the set p-use(i, j), is the set of variables for which there is a p-use for the edge (i, j). Examples: 11/16/99 CS 406 Testing

Set dcu(x, i) dcu(x, i) represents all those nodes in which the global c-use of x uses the value assigned by the def of x in i. Suppose a variable x is in def(i) of node i. Set of nodes, such that : Each node has x in its c-use There is a def-clear path from i to j Examples: 11/16/99 CS 406 Testing

Set dpu(x, i) dpu(x, i) represents all those edges in which the p-use of x uses the value assigned by the def of x in i. Suppose a variable x is in def(i) of node i. Set of edges, such that : each edge has x in its p-use There is a def-clear path from i to (j, k) 11/16/99 CS 406 Testing

Test case selection criteria Let G be the def-use graph for a program Let P be a set of all the complete paths of G that were executed during the test. Examples: 11/16/99 CS 406 Testing

All-defs criterion Make sure that the definitions of all variables are tested. For every def of every variable, one of its uses (either p-use or c-use) must be included in a path. For every node i in G and every x in def(i), P includes a def-clear path w.r.t. x to some member of dcu(x, i) 11/16/99 CS 406 Testing

All-p-uses criterion All p-uses of all definitions should be tested. For every x  def (i), P includes a def-clear path w.r.t. x from i to all members of dpu(x, i). Note that a c-use may not be tested Can require all p-uses, some c-uses criterion 11/16/99 CS 406 Testing

All c-uses criterion All c-uses of all definitions should be tested. For every x  def (i), P includes a def-clear path w.r.t. x from i to all members of dcu(x, i). Note that a p-use may not be tested Can require all c-uses, some p-uses criterion 11/16/99 CS 406 Testing

All-uses criterion All p-uses and all c-uses of a definition must be exercised. The set P must include, for every node i and every x  def(i) a def-clear path w.r.t. x from i to all elements of dcu(x, i) and to all elements of dpu(x, i). 11/16/99 CS 406 Testing

Number of test cases Can be shown theoretically that the limit of the size of test set is up to quadratic in the number of two-way decision statements in the program. Empirical studies have shown that the actual number of test cases that satisfy a criterion is quite small and increases linearly with the number of two-way decisions. 11/16/99 CS 406 Testing

Inclusion relationship between criteria all-paths (path coverage) all-uses all-c-uses/ some-p-uses all-p-uses/ some-c-uses all-defs all-p-uses all-edges (branch coverage) all-nodes (statement coverage) 11/16/99 CS 406 Testing

Implication of inclusion Suppose C1 includes C2 (C1  C2), I.e test cases satisfying C1 also satisfy C2 Does it mean that C1 is always better than C2? Statistically, the set of test cases satisfying C1 will be better than those satisfying C2. Testing done using these criteria performs better than randomly selected test cases. 11/16/99 CS 406 Testing

Test assessment Develop Question: a test set T a collection of test inputs Question: How good is T? Test assessment is the measurement of the goodness of T. Test assessment is carried out based on one or more criteria. 11/16/99 CS 406 Testing

Test adequacy assessment These criteria are known as test adequacy criteria. Test assessment is also known as test adequacy assessment. 11/16/99 CS 406 Testing

Test adequacy assessment (contd) Test adequacy assessment provides the following information: A metric, also known as adequacy score, or coverage, usually between 0 and 1. A list of all the weaknesses found in T, which when removed, will raise the score to 1. The weaknesses depend on the criteria used for assessment. 11/16/99 CS 406 Testing

Test adequacy assessment (contd) Once the coverage has been computed, and the weaknesses identified, one can improve T. Improvement of T is done by examining one or more weaknesses and constructing new test requirements designed to overcome the weaknesses. 11/16/99 CS 406 Testing

Test adequacy assessment (contd) This is continued until all weaknesses are overcome, i.e., the adequacy criterion is satisfied (coverage = 1) In some instances, it may not be possible to satisfy the adequacy criteria for one or more of the following reasons: Lack of sufficient resources (people, time) Weaknesses that cannot be removed because they are infeasible The cost of removing the weakness is not justified 11/16/99 CS 406 Testing

Test adequacy assessment (contd) While improving T by removing its weaknesses, one usually tests the program more thoroughly than it has been tested so far. This additional testing is likely to result in the discovery of remaining errors. Hence, we say that test assessment and improvement helps in the improvement of software reliability. 11/16/99 CS 406 Testing

Test adequacy assessment - summary procedure 0. Develop T 1. Select an adequacy criterion C 2. Measure adequacy criterion of T w.r.t C 3. Is T adequate? Yes No Yes 4. Improve T 5. More testing is warranted? No 6. DONE!! 11/16/99 CS 406 Testing