1. Introduction to SoftwareTesting
CSC 450
Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc.

7. What is software testing?
“Testing is the process of exercising a program with the intent of finding Errors.”
“Testing is the process of exercising a program with the intent of proving its correctness.”

8. What is software testing?
“Observing the execution of a software system to validate whether it behaves as intended and identify potential malfunctions.”
Antonia Bertolino

9. What does testing show? errors requirements conformance performance
an indication
of quality

10. What does testing NOT show?
Bugs remaining in the code! …

12. Why is testing so difficult?1 3 2 5 4
loop < =20 X a b c d e
How many paths are there through this loop?
How long would it take to execute all 1 test per millisecond (10-3)?

13. A Testing Scenario There are about 1014 possible paths3 2 5 4
loop < =20 X a b c d e
There are about 1014 possible paths
How long would it take to execute all paths?

14. A Testing Scenario 1 3 2 5 4
loop < =20 X a b c d e
How long would it take to execute 1014 possible 1 paths per 10-3 seconds?
1 year =31,536,000 seconds ~= 32,000,000,000 milliseconds
Total time = 1014 / 32 * 109 ~= 105 /32 years
About 3,170 one test/millisecond
About 3 one test/microsecond
About 11 one test/nanosecond

15. Why is testing so difficult?1 3 2 5 4
loop <= 20 X a b c d e
Because exhaustive testing is impossible
Exhaustive testing means:
Testing a program with all combinations of input values and preconditions for each element of the software under test and/or
Testing all paths through a program.

16. Why is testing so difficult?
Looping :
for (i=0; i< n; i++) etc.
Loop header
Conditional
statement
statement
How long would it take to execute all 1 test per millisecond (10-3)?

17. Terms! Terms! Terms!
Anomaly
BUG
Failure
Error
Defect
Fault
Exception

18. Testing Terminology Fault/defect Error Failure
An incorrect step, process, condition, or data definition in a computer program.
Software problem found after release (Pressman)
Error
An incorrect internal state resulting from a fault.
A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition (ISO).
Software problem found before release (Pressman)
Failure
The inability of a system or component to perform its required functions within specified performance requirements (IEEE).
Failures are caused by faults, but not all faults cause failures.

19. Testing Terminology Anomaly Bug Exception
Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents (IEEE).
Bug
Things the software does that it is not supposed to do
Something the software doesn't do that it is supposed to.
Exception
An event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).

20. Testing Terminology: Testing vs. Debugging
Evaluating software to determine conformance to some objective.
Debugging
Finding a fault, given a failure.

21. Testing Terminology: The RIP Fault/Failure Model
Three conditions are necessary for failures to be observable.
Reachability: The location of the fault must be reached.
Infection: After executing the location, the state of the program must be incorrect.
Propagation: The infected state must propagate to cause incorrect program output.

22. Testing Terminology Validation Verification
Does software meet its goals? Are we building the right product? User-centric!
Verification
Are we building the product right? E.g., does the code reflect the design? Developer-centric!
Name for the artifact being tested
Implementation under test (IUT)
Method under test (MUT), object under test (OUT)
Class/component under test (CUT), etc.

23. Testing Terminology: Test Plan
A test plan specifies how we will demonstrate that the software is free of faults and behaves according to the requirements specification
A test plan breaks the testing process into specific tests, addressing specific data items and values
Each test has a test specification that documents the purpose of the test
If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests
The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results

24. Testing Terminology: Test Oracle
A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testing
Test oracles are extremely difficult to create and are ideally created from the requirements specification

25. Testing Terminology: Test Case
A test case is a specification of
The pretest state of the IUT
Prefix values: values needed to put software into state required before test execution.
A set of inputs to the system
Postfix values: values that need to be sent to the software after test execution.
The expected results

26. Testing Terminology Test Point Test suite Domain
A specific value for test case input and state variables.
Domain
Set of values that input or state variables of IUT may take.
Heuristics for test point selection
Equivalence classes – one values in set represents all values in set.
Partition testing techniques
Boundary value analysis
Special values testing
Test suite
A collection of test cases.

27. Testing Terminology Fault model Test strategy Test model Test design
A description about where faults are likely to occur in a program.
Test strategy
An algorithm or heuristic for creating test cases from a representation, an implementation or a test model
Test model
A description of relationships between elements of a representation or implementation.
Test design
The process of creating a test suite using a test strategy.
Test effectiveness: relative ability of testing strategy to find bugs.
Test efficiency: relative cost of finding bugs.

28. Testing Terminology: Testing Strategies
Responsibility-based test design
Behavioural, functional, black-box, etc. testing
Implementation-based test design
Relies on source code,, e.g., white-box
Fault-based testing

29. A Classification of Testing: The V Model
Requirements Analysis
Acceptance Test
Architectural Design
System Test
Subsystem Design
Integration Test
Detailed Design
Module Test
Implementation
Unit Test

30. A Classification of Testing
Graphs
Logical expressions
Input domain characterizations
Syntactic descriptions

31. Test-Case Exercise
UML class model for Figure Hierarchy

32. The Polygon Class abstract class Polygon {
abstract void draw(int r, int g, int b); /* color closed area*/
abstract void erase(); /* set to background rgb */
abstract float area(); /* return area */
abstract float perimeter(); /* return sum of sides */
abstract Point center(); /* return centroid pixel */ }

33. The Triangle Class class Triangle extends Polygon {
public Triangle(LineSegment a, LineSegment b, LineSegment c){…}
public void setA(LineSegment a) {…}
public void setB(LineSegment b) {…}
public void setC(LineSegment c) {…}
public LineSegment getA() {…}
public LineSegment getB() {…}
public LineSegment getC() {…}
public boolean is_isoceles(){…}
public boolean is_scalene(){…}
public boolean is_equilateral(){…}
public void draw(int r, int g, int b){…}
public void erase(){…}
public float area(){…}
public float perimeter(){…}
public Point center(){…} }

34. The LineSegment Class class LineSegment extends Figure {
public LineSegment(Point x1, Point y1, Point x2, Point y2){…}
public void setx1(Point x1) {…}
public void sety1(Point y1) {…}
public void setx2(Point x2) {…}
public void sety2(Point y2) {…}
public Point getx1() {…}
public Point gety1() {…}
public Point getx2() {…}
public Point gety2() {…} }

35. Test-Case Exercise
Specify as many test cases as you can for the Triangle class defined above.
The class is used to determine if a triangle is: isosceles, equilateral or scalene.
A triangle is isosceles if two of its sides are equal.
A triangle is equilateral if all its sides are equal.
A triangle is scalene if its sides are of different sizes.

36. Test-Case Exercise A valid triangle must meet two conditions:
No side may have a length of zero.
Each side must be smaller than the sum of all sides divided by two.
i.e. Given s = (a + b + c)/2, then a < s, b < s, c < s must hold.
Stated differently, a < s && b < s && c < s  a < b + c, b < a + c and c < a + b.

37. Test-Case Exercise
Specify as many test cases as you can for the Triangle class defined above.
Test Case Description
Test Case Input a b c
Expected Output 1
Valid isosceles triangle 3 4
Isosceles 2 28 29 30 31 32 33 34 35

38. Solution To Test Case Exercise

39. Solution To Test Case Exercise

40. Solution To Test Case Exercise

41. Solution To Test Case Exercise

42. The C++ IntSet Class Test Case Design Exercise 2
Design test cases to test the methods (ignore the constructor and destructor) of the IntSet class.
The class performs operations on a single set – duplications are not allowed
If add(val) is called and val is already in the set, then the Duplicate exception is thrown.
class IntSet {
public:
//operations on a set of integers
IntSet(); /* constructor */
~IntSet(); /* destructor */
IntSet& add (int); /* add a member */
IntSet& remove (int); /* remove a member */
IntSet& clear (int); /* remove all members */
int isMember (int arg); /* Is arg a member */
int extent (); /* Number of elements */
int isEmpty (); /* Empty or not */ }
Design test cases to test the methods (ignore the constructor and destructor).

43. Who Tests the Software? developer independent tester
Understands the system
but, will test "gently“
and, is driven by "delivery"
Must learn about the system
but, will attempt to break it
and, is driven by quality

44. Testing Strategy
We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’
For conventional software
The module (component) is our initial focus
Integration of modules follows
For OO software
“testing in the small”  OO class
Has attributes and operations and implies communication and collaboration

45. Junit Overview

46. Unit Testing module to be tested results software engineer test cases
interface
local data structures
boundary conditions
independent paths
error handling paths
The module being tested should be reviewed in context of the requirements specification
The units comprising a system are individually tested
The code is examined for faults in algorithms, data and syntax
A set of test cases is formulated and input and the results are evaluated

47. Unit Test Environment test cases RESULTS driver Module stub stub
interface
local data structures
Module
boundary conditions
independent paths
error handling paths
stub
stub
test cases
RESULTS

48. What is JUnit?
JUnit is an open source unit testing framework for Java created by Kent Beck and Erich Gamma.
JUnit features include:
Assertions for testing expected results
Test fixtures for sharing common test data
Test suites for easily organizing and running tests
Graphical and textual test runners

49. JUnit Terminology
Fixture: a set of objects against which tests are run
A test fixture is ideal when two or more tests are to be executed for a common set of objects. In these scenarios, a test fixture avoids duplicating the initialization tasks before each test and the cleanup activities after each test
setup: a method which sets up the fixture, called before each test is executed.
teardown: a method to tear down the fixture, called after each test is executed.
Test Case: a class which defines the fixture to run multiple tests
- create a subclass of TestCase
- add an instance variable for each part of the fixture
- override setUp() to initialize the variables and allocate resource
- override tearDown() to release any permanent storage, etc.
Test Suite: a collection of test cases.

50. The JUnit Framework

51. JUnit 3

52. How To Write JUnit 3 Test Cases
Import the Junit framework:
import junit.framework.*;
Extend the “TestCase” class
extend junit.framework.TestCase;
Make use of special method-name conventions:
Begin each test method name with the prefix ‘test’.
Each test case is written in a method that has a name that begins with “test”
Create a mehod with the name setUp to initialize your object, etc.
Create a mehod with the name tearDown to clean-up after your test.
etc.

53. A Case Study class Money { private int fAmount;
private String fCurrency;   
public Money(int amount, String currency) {  
fAmount= amount;        
fCurrency= currency;     }
public int amount() {         return fAmount;     }
   
public String currency() {         return fCurrency;     }
public Money add(Money m) {    
return new Money(amount()+m.amount(), currency()); } }

54. How to Write A Simple Standalone TestCase
import junit.framework.*;
public class MoneyTest extends TestCase {    
//…    
public void testSimpleAdd() {        
Money m12 = new Money(12, "Dollar");  // (1)        
Money m14 = new Money(14, "Dollar");
                
Money expected = new Money(26, "Dollar");
       
Money result = m12.add(m14);    // (2)
Assert.assertTrue(expected.equals(result));     // (3)     }
Creates the objects we will interact with during the test. This testing context is commonly referred to as a test's fixture. All the fixture contains for the testSimpleAdd test are some Money objects.
Exercises the objects in the fixture.
Verifies the result

55. Assert assertEquals(expected, actual)
assertEquals(message, expected, actual)
assertEquals(expected, actual, delta)
assertEquals(message, expected, actual, delta)
assertFalse(condition)
assertFalse(message, condition)
AssertNotNull(object)
AssertNotNull(message, object)
AssertNotSame(expected, actual)
AssertNotSame(message, expected, actual)
assertTrue(condition)
assertTrue(message, condition)

56. Start to Use it 2. Installation unzip the junit.zip file
1. Download the latest version of JUnit from
2. Installation
unzip the junit.zip file
add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar
3. Testing
Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK.
for the batch TestRunner type:     java junit.textui.TestRunner junit.samples.AllTests
for the graphical TestRunner type:     java junit.awtui.TestRunner junit.samples.AllTests
for the Swing based graphical TestRunner type:     java junit.swingui.TestRunner junit.samples.AllTests
Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path
Important: Don't install the junit.jar into the extension directory of your JDK installation.
If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse

57. A Second Example import junit.framework.*;
public interface ISorting {
//etc. }
public SortingComparison implements ISorting
{ //... }
public void bubbleSort(int[] a); //implements bubblesort algorithm
import junit.framework.*;
public class SortingTest extends TestCase {
SortingComparison obj;//instance variable in the fixture
public SortingTest(String name) {
super(name); }
public void testBubble() {
//SortingComparison obj = new SortingComparison();
int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0};
int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000};
obj.bubbleSort(qArray);
obj.printArray(qArray);
obj.printArray(rArray);
assertTrue(java.util.Arrays.equals(qArray, rArray)); }
public void setUp() {
obj = new SortingComparison(); }
public void testSimpleTest() {
int answer = 2;
assertEquals((1+1), answer); }
public void tearDown () {
//do something }

58. Example code 1 import junit.framework.TestCase;
public class JunitExamples extends TestCase {
SortingComparison obj;//instance variable in the fixture
public JunitExamples(String name) {
super(name); }
public void setUp() {
obj = new SortingComparison();
public void tearDown () {
//do something
public void testBubble(){
//SortingComparison obj = new SortingComparison();
int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0};
int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000};
obj.bubbleSort(qArray);
obj.printArray(qArray);
obj.printArray(rArray);
assertTrue(java.util.Arrays.equals(qArray, rArray));
}//end of method
}//end of class

59. Example code 2 import junit.framework.TestCase;
public class JunitExamples extends TestCase {
SortingComparison obj;//instance variable in the fixture
public JunitExamples(String name) {
super(name); }
public void setUp() {
obj = new SortingComparison();
public void tearDown () {
//do something
public void testIsosceles (){
int side1 =3, side2 = 3, side3 = 0;
boolean result = is_isosceles(side1, side2, side3);
assertTrue(result == true);
}//end of method}//end of class

60. Running JUnit java junit.textui.TestRunner JavaClassfileName
java junit.swingui.TestRunner JavaClassfileName
java junit.awtui.TestRunner JavaClassfileName
Use eclipse
Download the JUnit and integrate in eclipse
Create your java project and the java application to be tested.
then … (next slide)

61. This will be filled in automatically
JUnit in Eclipse
To create a test class, select File New Other...  Java, JUnit, TestCase and enter the name of the class you will test
Fill this in
This will be filled in automatically

62. Running JUnit Second, use this pulldown menu
First, select a Test class
Third, Run As  JUnit Test

63. Results
Your results are here

64. Start to Use it 2. Installation unzip the junit.zip file
1. Download the latest version of JUnit from
2. Installation
unzip the junit.zip file
add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar
3. Testing
Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK.
for the batch TestRunner type:     java junit.textui.TestRunner junit.samples.AllTests
for the graphical TestRunner type:     java junit.awtui.TestRunner junit.samples.AllTests
for the Swing based graphical TestRunner type:     java junit.swingui.TestRunner junit.samples.AllTests
Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path
Important: Don't install the junit.jar into the extension directory of your JDK installation.
If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse

65. JUnit 4

66. Comparing JUnit 3 to JUnit 4
All the old assertXXX methods are the same
Most things are about equally easy
JUnit 4 makes it easier to test that exceptions are thrown when they should be
JUnit 4 can still run JUnit 3 tests
JUnit 4 provides protection against infinite loops
JUnit 4 has some additional features

67. Migrating to JUnit 4 JUnit 4 requires Java 5 or newer
Don’t extend junit.framework.TestCase; just use an ordinary class
Import org.junit.* and org.junit.Assert.*
Use a static import for org.junit.Assert.*
Static imports replace inheritance from junit.framework.TestCase
Use annotations instead of special method names:
Instead of a setUp method, before some method
Instead of a tearDown method, before some method
Instead of beginning test method names with ‘test’, before each test method

68. Testing Strategies

69. Testing Strategy unit test integration test Black Box Testing
White Box Testing
system
test
validation
test

70. Unit Testing module to be tested
software
engineer
results
module
to be
tested
test cases
interface
local data structures
boundary conditions
independent paths
error handling paths
The module being tested should be reviewed in context of the requirements specification
The units comprising a system are individually tested
The code is examined for faults in algorithms, data and syntax
A set of test cases is formulated and input and the results are evaluated

71. Unit Test Environment test cases RESULTS driver Module stub stub
interface
local data structures
Module
boundary conditions
independent paths
error handling paths
stub
stub
test cases
RESULTS

72. Integration Testing
The goal is to ensure that groups of components work together as specified in the requirements document
Four kinds of integration tests exist
Structure tests
Functional tests
Stress tests
Performance tests

73. Integration Testing Strategies
Options:
• the “big bang” approach
• an incremental construction strategy

74. Top Down Integration A top module is tested with stubs B F G
stubs are replaced one at
a time, "depth first" C
as new modules are integrated,
some subset of tests is re-run D E

75. Bottom-Up IntegrationF G
drivers are replaced one at a
time, "depth first" C
worker modules are grouped into
builds and integrated D E
cluster

76. Sandwich Testing cluster A Top modules are tested with stubs B F G C
Worker modules are grouped into
builds and integrated D E
cluster

77. System Testing
The goal is to ensure that all elements that interact to solve the problem do so correctly.
Software, databases, hardware, people, procedures.
Overall functionality and performance must be achieved.

78. Validation Testing
The goal is to ensure that the system actually does what the customer expects it to do – fulfils requirements
Testing is carried out by customers mimicking real world activities
Customers should also intentionally enter erroneous values to determine the system behavior in those instances

79. Black Box Testing Testing the functionality of the program
The tester either knows nothing about the internal structure of the code or otherwise ignores this information
Test cases are formulated based on expected output of methods
Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behaviour is the same

80. White Box Testing
The tester uses knowledge of the programming constructs to determine the test cases to use
If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations
Similarly, conditions would be tested for true and false

81. High Order Testing Validation testing System testing
Focus is on software requirements
System testing
Focus is on system integration
Alpha/Beta testing
Focus is on customer usage
Recovery testing
forces the software to fail in a variety of ways and verifies that recovery is properly performed
Security testing
verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
Stress testing
executes a system in a manner that demands resources in abnormal quantity, frequency, or volume
Performance Testing
test the run-time performance of software within the context of an integrated system

82. Strategic Issues State testing objectives explicitly.
Understand the users of the software and develop a profile for each user category.
Develop a testing plan
Build “robust” software that is designed to test itself
Use effective formal technical reviews as a filter prior to testing
Conduct formal technical reviews to assess the test strategy and test cases themselves.
Develop a continuous improvement approach for the testing process.

83. Testing Steps Determine what the test is supposed to measure
Decide how to carry out the tests
Develop the test cases
Determine the expected results of each test (test oracle)
Execute the tests
Compare results to the test oracle

84. Object-Oriented Testing
begins by evaluating the correctness and consistency of the OOA and OOD models
testing strategy changes
the concept of the ‘unit’ broadens due to encapsulation
integration focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario
validation uses conventional black box methods
test case design draws on conventional methods, but also encompasses special features

85. Broadening the View of “Testing”
“It can be argued that the review of OO analysis and design models is especially useful because the same semantic constructs (e.g., classes, attributes, operations, messages) appear at the analysis, design, and code level. Therefore, a problem in the definition of class attributes that is uncovered during analysis will circumvent side effects that might occur if the problem were not discovered until design or code (or even the next iteration of analysis)” - Pressman.

86. OOT Strategy class testing is the equivalent of unit testing
operations within the class are tested
the state behavior of the class is examined
integration applied three different strategies
thread-based testing—integrates the set of classes required to respond to one input or event
use-based testing—integrates the set of classes required to respond to one use case
cluster testing—integrates the set of classes required to demonstrate one collaboration

87. Smoke Testing
A common approach for creating “daily builds” for product software
Smoke testing steps:
Software components that have been translated into code are integrated into a “build.”
A build includes all data files, libraries, reusable modules, and engineered components that are required to implement one or more product functions.
A series of tests is designed to expose errors that will keep the build from properly performing its function.
The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind schedule.
The build is integrated with other builds and the entire product (in its current form) is smoke tested daily.
The integration approach may be top down or bottom up.

88. Debugging: A Diagnostic Process

89. The Debugging Process Debugging test cases results new test cases
regression
tests
suspected
causes
corrections
Debugging
identified
causes

90. Debugging Effort time required to diagnose the symptom and
determine the
cause
time required
to correct the error
and conduct
regression tests

91. Symptoms & Causes symptom cause symptom and cause may be
geographically separated
symptom may disappear when
another problem is fixed
cause may be due to a
combination of non-errors
cause may be due to a system
or compiler error
cause may be due to
assumptions that everyone
believes
symptom
cause
symptom may be intermittent

92. Consequences of Bugs infectious damage catastrophic extreme serious
disturbing
annoying
mild
Bug Type
Bug Categories:
function-related bugs,
system-related bugs, data bugs, coding bugs,
design bugs, documentation bugs, standards
violations, etc.

93. Debugging Techniques brute force / testing backtracking
Cause-elimination
Induction (hypothesis cause)
deduction (cause  hypothesis)

94. Debugging: Final Thoughts
symptom you're seeing.
Don't run off half-baked,
think
about the 1.
more insight.
Use tools
(e.g., dynamic debugger) to gain 2.
If at an impasse,
get help
from someone else. 3.
Be absolutely sure to
conduct regression tests
when you do "fix" the bug. 4.

95. White-Box Testing & Control Flow Graphs

96. Testing Strategy White Box Testing unit test integration test
Black Box Testing
system
test
validation
test

97. White-box testing: control-flow graph
Directed graph?
A control flow graph of a program is a graph that shows the paths that may be traversed when the program is executed.
Vertices are statements in the program

98. Control-flow Graph int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else3 4 5 6 7 8
int n = 0;
do{
if(n++ % 2 == 1)
dBase[i]++;
else
if(dBase[i] > 100)
dBase[i] -= n;
dBase[i] += n;
}while(n < 20); 1 2 4 3 5 6 7

99. Cyclomatic Complexity – V(G)
If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by:
V(G) = e – n + 2
Or V(G) = the number of decision nodes + 1

100. Cyclomatic Complexity – V(G)1 2 3 4 5 6 7 8
cyclomatic complexity
V(G) = e – n + 2
= 11 – 7 = 4
Or V(G) = the number of decision nodes + 1
= = 4

101. Cyclomatic Complexity
A number of industry studies have indicated
that the higher V(G), the higher the probability
or errors.
modules
V(G)
modules in this range are
more error prone

102. Cyclomatic Complexity – V(G)1 2 3 4 5 6 7 8
The cyclomatic complexity is a measure of the number of linearly independent paths in the graph.
An independent path must move along at least one edge not previously traversed
What are the independent paths for this graph? …8

103. White-box testing: Code coverage
White box testing methods emphasize code coverage
Statement coverage
Every statement is executed at least once
Branch coverage
Every branch is exercised at least once
i.e. every path from a node is traversed at least once
Once true and once false evaluation
Multiple condition coverage
All combinations of simple conditions are exercised
Basis-path model
Exercise V(G) independent paths
Data flow coverage
Evaluation of state of instance variables.

104. Statement Coverage 1 2 3 4 5 6 7 8
Write enough test cases to execute each statement al least once,
May miss bugs for complex predicates and loops

105. Statement Coverage int num = 0; do { }while(num != getValue(num));1 2 3 4 5 6 7 8
int num = 0;
do {
if(num > 5 && (sum-num)<findMax(num)) {
if(transform(num))
transpose(Math.pow(num, 2))
else
sum += num + 5; }
sum += findMax(num);
}while(num != getValue(num)); 1 2 3 6 5 4 7

106. Branch Coverage Determine number of branches1 2 3 4 5 6 7 8
Determine number of branches
Compute path for each branch
Write test cases to exercise each path

107. Multiple Condition Coverage1 2 3 4 5 6 7 8
Determine true/false combinations for conditions
Write test cases to exercise each combination
The required test cases may be determined using the truth table for the logical operator of the condition.

108. Multiple Condition Coverage
if(num > 5 || isValid(num) && (sum-num<findMax(num) || sum = 100) && isDivisible(num, 7)
Test Case #
num > 5
isValid(num)
(sum-num) < findMax(num)
sum=100
isDiv (num, 7) 1 F 2 T 3 4 5 6 7 8 9 10 11

109. Basis-path Testing Compute the cyclomatic complexity1 2 3 4 5 6 7 8
Compute the cyclomatic complexity
Derive independent paths
…7-8
Write test cases to exercise each path

110. Example 1 V(G) = 2 + 1 = 3 Statement coverage requires 2 test cases
Branch coverage requires 2 test cases
Basis-path coverage requires 3 test cases
There are four distinct entry-exit paths
Which should be chosen? 1 2 4 3 5 6 7

111. Example 2 A D
What is V(G)?
How many test cases are required for statement coverage?
How many test cases are required for branch coverage?
How many test cases are required for basis-path coverage?
How many unique entry-exit paths exist?
LIST THE PATHS F E K L

112. Introduction to Mutation Testing & Program Perturbation

113. What is Mutation Testing?
Mutation Testing is a testing technique that focuses on measuring the adequacy of test cases.
Mutation Testing is NOT a testing strategy like path or data-flow testing. It does not outline test data selection criteria.
Mutation Testing should be used in conjunction with traditional testing techniques, not instead of them.

114. Mutation Testing
Faults are introduced into a program by creating many versions of the program
Each version is called a mutant.
Each mutant contains a single fault.
Test cases are applied to the original program and to the mutant program.
The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.

115. Example of a Program Mutation
1 int max(int x, int y)
2 {
3 int mx = x;
4 if (x > y)
5 mx = x;
6 else
7 mx = y;
8 return mx;
9 }
1 int max(int x, int y)
2 {
3 int mx = x;
4 if (x < y)
5 mx = x;
6 else
7 mx = y;
8 return mx;
9 }

116. Mutation Testing Process
Use mutation operators to introduce errors into the source program
One error per mutant
Validate the test suite with unmodified program (insures the test suite functions properly)
Run test suite on mutant programs
Analyze result to determine how many non-equivalent mutants were killed
Fix test suite to kill all live mutants until Mutation Score = 1.0 or specified a threshold score is reached e.g. 0.96

117. Mutation Testing Process
Process (cont.)
Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3

118. Test Case Adequacy
A test case is adequate if it is useful in detecting faults in a program.
A test case can be shown to be adequate by finding at least one mutant program that generates a different output than does the original program for that test case.
If the original program and all mutant programs generate the same output, the test case is inadequate.

119. Terminology Mutation testing Mutation
The creation of a set of mutant programs of the program being tested.
Mutation
a single syntactic change that is made to a program statement.
Each mutant differs from the original program by one mutation.

120. Terminology Mutant Mutant Killed Mutation Score Mutation Operator
A faulty program; the original with one fault introduced
Mutant Killed
Test suite detects the mutant
Mutation Score
#of killed mutants/ # of non-equivalent mutants
Mutation Operator
The function that changes the source code to create a mutant

121. Mutation Operators
Replace each operand with every other syntactically correct operand
Modify expressions by replacing operators and inserting new operators
Delete entire statements

122. Categories of Mutation Operators
Operand Replacement Operators:
Replace a single operand with another operand or constant. E.g.,
if(x > y)
if (5 > y) Replacing x by constant 5.
if (x > 5) Replacing y by constant 5.
if (y > x) Replacing x and y with each other.
E.g., if all operators are {+,-,*,**,/} then the following expression a = b * (c - d) will generate 8 mutants:
4 by replacing *
4 by replacing -.

123. Categories of Mutation Operators
Expression Modification Operators:
Replace an operator or insert new operators. E.g.,
if (x == y)
if (x >= y) Replacing == by >=.
if (x == ++y) Inserting ++.

124. Categories of Mutation Operators
Statement Modification Operators:
E.g.,
Delete the else part of the if-else statement.
Delete the entire if-else statement.
Replace line 3 by a return statement.
1 int max(int x, int y)
2 {
3 int mx = x;
4 if (x > y)
5 mx = x;
6 else
7 mx = y;
8 return mx;
9 }
return x;

125. Mutation System Example
The Mothra mutation system for FORTRAN77 supports 22 mutation operators. E.g.,
absolute value insertion
constant for array reference replacement
GOTO label replacement
RETURN statement replacement
statement deletion
unary operator insertion
logical connector replacement
comparable array name replacement

126. Why Does Mutation Testing Work?
The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.

127. The Competent Programmer Hypothesis
Programmers are generally very competent and do not create “random” programs.
For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program.
An incorrect program can be created from a correct program by making some minor change to the correct program.

128. Mutation Testing Algorithm
Generate program test cases.
Run each test case against the original program.
If the output is incorrect, the program must be modified and re-tested.
If the output is correct go to the next step ...
Construct mutants
E.g. using a tool like Mothra.

129. Mutation Testing Algorithm (Cont’d)
Execute each test case against each alive mutant.
If the output of the mutant differs from the output of the original program, the mutant is considered incorrect and is killed.
Two kinds of mutants survive:
Functionally equivalent to the original program: Cannot be killed.
Killable: Test cases are insufficient to kill the mutant. New test cases must be created.

130. Mutation Score
The mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test data.
Mutation Score = 100 * D / (N - E)
D = Dead mutants
N = Number of mutants
E = Number of equivalent mutants
A set of test cases is mutation adequate if its mutation score is 100%.

131. Evaluation
Theoretical and experimental results have shown that mutation testing is an effective approach to measuring the adequacy of test cases.
The major drawback of mutation testing is the cost of generating the mutants and executing each test case against them.
Very expensive to run (time, processor intensive, memory)

132. Program Perturbation
Program Perturbation is a technique to test a program’s robustness.
It is based on unexpectedly changing the values of program data during run-time.

133. Software Failure Hypothesis
Program perturbation is based on the three part software failure hypothesis:
Execution: The fault must be executed.
Infection: The fault must change the data state of the computation directly after the fault location.
Propagation: The erroneous data state must propagate to an output variable.

134. Program Perturbation Process
The tester must:
inject faults in the data state of an executing program;
trace the impact of the injected fault on the program’s output.
The injection is performed by applying a perturbation function that changes the program’s data state.

135. The Perturbation Function
The perturbation function is a mathematical function that:
takes a data state as its input;
changes the data state according to some specified criteria;
produces a modified data state as output.

136. The Fault Injection
A program location N is chosen along with a set of input variables I that are in scope at location N.
The program is executed until location N.
When execution arrives at location N, the resulting data state is changed (perturbed).
The subsequent execution will either fail or succeed.

137. Program Perturbation Example
Assume the following perturbation function:
1. int perturbation (int x)
2. {
3. int newX;
4. newX = x + 20;
5. return newX;
6. }

138. Example of a Fault Injection
1. main()
2. {
3. int x;
4. x = ReadInt();
4.1 x = perturbation(x);
5. if (x > 0)
printf(“X positive”);
7. else
printf(“X negative”);
9. }
1. main()
2. {
3. int x;
4. x = ReadInt();
5. if (x > 0)
6. printf(“X positive”);
7. else
8. printf(“X negative”);
9. }

139. What Perturbation Testing is and is Not
Perturbation testing is NOT a testing technique that outlines test selection and coverage criteria.
Rather, perturbation testing is a technique that can be used to measure the reliability of the software (tolerance to faults).

140. Evaluation
The program is repeatedly executed and injected with faults during each execution.
The ratio of the number of failures detected divided by the total number of executions is used to predict failure tolerance.

141. Summary & The End Qu es ti ons? ______________________
Devon M. Simmonds
Computer Science Department
University of North Carolina Wilmington
____________________________________________ _________________

142. Why is testing so difficult?1 3 2 5 4
loop < =20 X a b c d e
There are about 1014 possible paths
How long would it take to execute all paths?
About 3,170 one test/millisecond

143. Example 2 A
What is V(G)? - 7
How many test cases are required for statement coverage? - 4
How many test cases are required for branch coverage? - 4
How many test cases are required for basis-path coverage? - 7
How many unique entry-exit paths exist? – 16 D F E K L
ABDEGKMQS
ABDEGKNQS
ABDEHKMQS
ABDEHKNQS
ACDEGKMQS
ACDEGKNQS
ACDEHKMQS
ACDEHKNQS
ACDFJLPRS
ACDFJLQRS
ACDFILPRS
ACDFILQRS
ABDFJLPRS
ABDFJLQRS
ABDFILPRS
ABDFILQRS

144. Example 2 A D F E K L ABDEGKMQS ABDEGKNQS ABDEHKMQS ABDEHKNQS
ACDEGKMQS
ACDEGKNQS
ACDEHKMQS
ACDEHKNQS F E K L
ACDFJLPRS
ACDFJLQRS
ACDFILPRS
ACDFILQRS
ABDFJLPRS
ABDFJLQRS
ABDFILPRS
ABDFILQRS

145. Structure
setUp()
Storing the fixture's objects in instance variables of your TestCase subclass and initialize them by overriding the setUp method
tearDown()
Releasing the fixture’s
run()
Defining how to run an individual test case.
Defining how to run a test suite.
testCase()