Something to amuse you… CS UWO minutes

Find the problem (i.e. Is line/statement coverage enough?): CS UWO2 include main () { int x, pos, even; printf("Enter an integer:"); scanf("%d", &x); pos = 1; even = 1; if (x > 0) {pos = 1;} if (x%2 == 0) {even = 1;} printf (“Number: %d, Positive: %d, Even: %d\n", x, pos, even); return 0; } This code prints out 1 if the number is even and 1 if the number is positive and 0 for odd and 0 for negative. I tested it with a minimal test case (x=6) to get 100% line coverage (see image), and I got the correct result but still the program is wrong. WHY???

CS UWO3 Structural Testing Structural testing measures Flowcharts Structural testing and flowcharts Minimal test suites

CS UWO4 Program Flowcharts Example of code: Intended to set pos = 1 iff x is positive, even = 1 iff x is even Draw a graph of the statements and decisions in the code: Statements = boxes Decisions = diamonds This graph called a “flowchart” main () { int x, pos, even; printf("Enter an integer:"); scanf("%d", &x); pos = 0; even = 0; if (x > 0) {pos = 1;} if (x%2 == 0) {even = 1;} printf ("%d, %d, %d\n", x, pos, even); return 0; }

CS UWO5 Flowchart Example Printf() Scanf() X>0 Pos=1 X%2=0Even = 1 Printf() End T T Pos=0 Even=0 F F

CS UWO6 Flowchart Example A flowchart is really a labeled directed graph Boxes and diamonds = nodes Arrows = edges Edges indicate parts of “paths” which may be followed through code Labels on edges going out of diamonds indicate what happens when decision in diamond is true/false

CS UWO7 Flowcharts: Example 2 (Code) i = 0; while (s[i] != 0) { if (s[i] >= ‘a’) && (s[i] <= ‘z’)) { s[i]=s[i] – 32; } i++; }

CS UWO8 Flowcharts: Example 2 (Flowchart) i=0 i++ End s[i]=s[i]-32 s[i]!=0? s[i]>=‘a’&& s[i]<=‘z’? T T F F

CS UWO9 Structural Testing and Flowcharts What does this have to do with testing? If test suite executes all statements, then test suite visits every box and diamond in the flowchart We say the test suite covers every node We can also say the test suite covers every statement This does not mean that every arrow (edge) in the flowchart has been followed Example: consider the first example flowchart We run it on one test case: x=2 All nodes visited However, neither F branch has ever been taken If every arrow in the flowchart has been taken by some case in the test suite: We say the test suite covers every edge We can also say the test suite covers every decision

CS UWO10 Edge vs. Node Coverage Edge cover is more thorough than node coverage Catches more problems Example: consider first example flowchart Change statements pos =0; even = 0 in the first column to pos = 1; even = 1 Now, program always sets pos and even to 1 But we wanted pos to indicate whether x is positive, even to indicate whether x is even If we just test with x = 2

CS UWO11 Printf() Scanf() x>0 Pos=1 x%2=0Even = 1 Printf() End T T Pos=1 Even=1 F F Would 100% Statement/Node Coverage Find the Mistake with test case x=2? Coding mistake made by a novice programmer (initialized incorrectly)

CS UWO12 If we test with both x = 2 and x = -1: —We cover every edge —We also force a failure to happen, program reports wrong results for x = - 1 Thus, edge coverage can reveal failures where node coverage cannot.

CS UWO13 Coverage Terminology If a test suite executes 100% of statements in program, we say it achieves 100% node coverage or 100% statement coverage If a test suite follows 90% of edges in flowchart, we say it achieves 90% edge coverage (etc) “Line Coverage”: what gcov measures Any line containing code is measured —Considered covered if any code on line is executed Not exactly the same thing as statement coverage —Can have more than one statement on a line However, 100% statement coverage implies 100% line coverage Edge coverage also called decision coverage Tests each decision in an if, while, etc. both ways

CS UWO14 Strength of Coverage Measures 100% node coverage implies 100% line coverage However, 100% line coverage does not imply 100% node coverage 100% edge coverage implies 100% node coverage If followed every edge, must have visited every node However, as we saw, 100% node coverage does not necessarily imply 100% edge coverage Thus we say: Node (statement) coverage is strong than line coverage Edge (decision) coverage is stronger than node coverage We can draw a graph: Arrow from A to B: B is a stronger coverage measure than A Line Node (Statement) Edge (Decision)

CS UWO15 Question Draw a flowchart for the following c code and find the minimum number of test cases to achieve: 100% line coverage 100% node coverage 100% edge coverage int main () { int a, b; a = 6; printf("\n\nEnter a value for a: "); scanf ("%d",&a); printf("\n\nEnter a value for b: "); scanf ("%d", &b); if (( a 15)) if (b ==9) printf ("good\n"); else printf("evening\n"); else printf("madam"); printf("sir\n"); return 0; }

CS UWO16 Your Answer

CS UWO17 Minimal Test Suites By throwing more and more test cases into a test suite, we may achieve higher coverage However, longer test suites take longer to run Hence, we often want to find some minimal test suite Important to define what minimal means i.e. minimal test cases? Minimal number of input used? Minimum time to run? Example: first flowchart example: Test suite: x=2, x=1, x=-2 covers all edges However, it has 3 test cases Test suite: x=2, x=-1 covers al edges and has only 2 test cases Cannot have a minimal test suite which covers all edges and has only 1 test case —Have to execute each decision both ways Hence, x=2, x=-1 is a minimal test suite for 100% edge coverage Minimal in the sense of needing the fewest test cases

CS UWO18 Coverage and Test Suites: Another Example Consider again the case-conversion code flowchart (Example 2) Find “minimal” test suite which achieves: 100 % node coverage 100% edge coverage Decide what minimal means in this context!

CS UWO19 Case Conversion Flowchart: Coverage Clearly we can achieve 100% node coverage with one-char test case Say, for example: “a” This test case also follows both arrows from the top diamond This is because: —First time through loop (i==0), T branch taken —Next time condition is evaluated (i==1), s[i] is the null (end of string) character —Hence, F branch is also taken However, F branch of other diamond not taken To cover all edges, we need test suite like (A) s= “a”, s=“X”; or (B) s = “aX”

CS UWO20 Case Conversion Flowchart: Minimality Which of (A) and (B) is “minimal” Depends on how we define minimality (A) has 2 test cases, but each has just 1 character (B) has just 1 test case, but it has 2 characters We should always define precisely what we mean by minimality for a given test suite Here, makes more sense to accept (B) as minimal Don’t’ have to run program as many times, incur overhead of starting up process again Hence, for this problem, we should say something like: Test suite X will be considered smaller than test suite Y if either: —X has few test cases than Y; or —X has the same number of test cases as Y, but the total number of characters in X is smaller Saves us from having to accept s = “aaaaaaXXXXXX” as minimal

CS UWO21 Stronger Coverage Measures So far we have been writing compound decisions (e.g. (c >=‘a’) && (c <=‘z’)) inside a single diamond Decision: the whole thing in parentheses after the “if”, while”, etc Consider the code: if ((c>=‘a’) && (c<=‘z’)) {..} (c >=‘a’) && (c<=‘z’) is the decision Conditions: the individual terms of the condition, connected by logical operators (c>=‘a’) && (c<=‘z’) are the two conditions If we separate each condition with the decision into a separate diamond: We can get a better reflection of what the program does

CS UWO22 Splitting Up Diamonds: Flowcharts Example: Instead of writing we write Now we have two new arrows To cover all arrows, we have to cover the two ones as well s[i]>=‘a’ && s[i]<=‘z’ s[i]>=‘a’ s[i]<=‘z’ T T T F F F

CS UWO23 Short Circuit Condition Coverage If we split up all diamonds and then cover all arrows: We are achieving more thorough coverage than just edge (decision) coverage can give us This is called short circuit (SC) condition coverage Recall definition of short circuit evaluation —Evaluation of a Boolean expression is cut short as soon as it can be determined to be true or false Applies to language with short circuit evaluation of decisions, e.g. C, C++, Java Similar but not identical to another notion called just “condition coverage” Condition coverage applies to languages without short circuit evaluation of decisions, e.g. Visual Basic We will not study this coverage measure

CS UWO24 Achieving SC Condition Coverage Consider the case conversion example Test case “aX” is sufficient to achieve 100% edge coverage Whole decision (c>=‘a’) && (c<=‘z’) is tested both ways: true for ‘a’ and false for ‘X’ Is not sufficient to achieve 100% SC condition coverage Have not take the F arrow from the s[i] <= ‘z’ diamond To take that arrow we need a character that is: Greater than or equal to ‘a’ Greater than ‘z’ We need one such ASCII character The test case “aX~” will achieve 100% SC condition coverage

CS UWO25 Achieving SC Condition Coverage in General In general, for a decision A && B to achieve SC condition coverage, we need to design test cases to make: A true, B true A true, B false A false We will not even be able to get to the evaluation of B unless A is true as in first two cases (due to SC evaluation) Similarly, for a decision A || B to achieve SC condition coverage, we need to design test cases to make: A false, B true A false, B false A true We will not even be able to get to the evaluation of B unless A is false as in the first two cases

CS UWO26 SC Condition Coverage vs. Edge Coverage If we have achieved SC condition coverage: We must have evaluated each condition of an “if”, “while”, etc. both ways Therefore, we must have evaluated each decision both ways Therefore, SC condition coverage is stronger than edge (decision) coverage Is there anything stronger than SC condition coverage?

CS UWO27 Path Coverage “100% path coverage”: every possible path through the flowchart of the code has been executed by some test case Path: sequence of nodes visited during an execution —Can contain repeated elements if execution has visited a node more than once The strongest possible coverage measure For code with no loops, this is achievable For code with while loops: Going around loop a different number of times means we have taken a different path For some code, we can go around a loop any number of times Therefore, for some code: it is not possible to achieve 100% path coverage

CS UWO28 Path Coverage: Example 1 Integer classification example again: Four paths through the code are possible X = 2 X = 1 X = -2 X = -1 Draw the paths for each of the above examples! Therefore we need four test cases for 100% coverage Printf() Scanf() X>0 Pos=1 X%2=0Even = 1 Printf() End T T Pos=0 Even=0 F F

CS UWO29 Path Coverage: Example 2 Case conversion example again Every string of a different length will follow a different path i=0 i++ End s[i]=s[i]-32 s[i]!=0? s[i]>=‘a’&& s[i]<=‘z’? T T F F

CS UWO30 Path Coverage for Loops Path coverage for many while loops is impossible Most programs have while loops! Most have at least one that can be followed an arbitrary number of times: e.g. A top level loop where we read a command and execute it Therefore, for most whole programs, we don’t even attempt 100% path coverage Path coverage is still useful for small sections of code

CS UWO31 Strength of Coverage Measures Line (weakest) Node (Statement) Edge (Decision) Path (strongest) Where arrow from A to B means “B is stronger than A”

CS UWO32 Testing Loops If we have loops, we can’t test all possible paths However, just decision coverage may not spot usual kinds of errors Often code is wrong because programmer didn’t consider what would happen when: Loop decision is false right from the start Loop decision is true once, then false Sometimes there is a maximum number (max) of times loop can be executed Example: loop may stop at end of an array, or when some condition is true In these cases, code may be wrong because programmer didn’t consider what would happen when Loop decision is true max times Loop decision is true max-1 times

CS UWO33 Loop Coverage It is therefore useful to write test cases which: Execute loop 0 times Execute loop 1 time Execute loop more than 1 time Execute loop max times (if max exists) Execute loop max -1 times (if max exists) These tests are often referred to as loop coverage However, there is no real formal definition of loop coverage as with node, edge etc. coverage

CS UWO34 Loop Coverage: Exercise Provide adequate loop coverage for the following: (code finds the rightmost dot in a string, used to find extension for filename) i = strlen(fname) -1; while (i>0 && fname[i] != ‘.’) {i--;} To do “loop coverage”, list some of the possible test cases: (0 times)  (1 time)  (more than 1 time)  (max – 1 times)  (max times)  Note: max here is the length of the string minus 1 Note: the code above is almost certainly correct However, problem may not be with the loop itself, but the code that comes after For instance, maybe the code that comes afterward: —Expects a non-empty extension —Expects a non-empty filename before the extension —Can’t handle the case of no extension Loop testing will help find those problems too fname contains: “report1.” fname contains: “report1.t” fname contains: “report1.txt” fname contains: “.txt” fname contains: “report1”