Download presentation
Presentation is loading. Please wait.
1
Introduction to SoftwareTesting
CSC 450 Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc.
7
What is software testing?
“Testing is the process of exercising a program with the intent of finding Errors.” “Testing is the process of exercising a program with the intent of proving its correctness.”
8
What is software testing?
“Observing the execution of a software system to validate whether it behaves as intended and identify potential malfunctions.” Antonia Bertolino
9
What does testing show? errors requirements conformance performance
an indication of quality
10
What does testing NOT show?
Bugs remaining in the code! …
12
Why is testing so difficult?
1 3 2 5 4 loop < =20 X a b c d e How many paths are there through this loop? How long would it take to execute all 1 test per millisecond (10-3)?
13
A Testing Scenario There are about 1014 possible paths
3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths?
14
A Testing Scenario 1 3 2 5 4 loop < =20 X a b c d e How long would it take to execute 1014 possible 1 paths per 10-3 seconds? 1 year =31,536,000 seconds ~= 32,000,000,000 milliseconds Total time = 1014 / 32 * 109 ~= 105 /32 years About 3,170 one test/millisecond About 3 one test/microsecond About 11 one test/nanosecond
15
Why is testing so difficult?
1 3 2 5 4 loop <= 20 X a b c d e Because exhaustive testing is impossible Exhaustive testing means: Testing a program with all combinations of input values and preconditions for each element of the software under test and/or Testing all paths through a program.
16
Why is testing so difficult?
Looping : for (i=0; i< n; i++) etc. Loop header Conditional statement statement How long would it take to execute all 1 test per millisecond (10-3)?
17
Terms! Terms! Terms! Anomaly BUG Failure Error Defect Fault Exception
18
Testing Terminology Fault/defect Error Failure
An incorrect step, process, condition, or data definition in a computer program. Software problem found after release (Pressman) Error An incorrect internal state resulting from a fault. A discrepancy between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition (ISO). Software problem found before release (Pressman) Failure The inability of a system or component to perform its required functions within specified performance requirements (IEEE). Failures are caused by faults, but not all faults cause failures.
19
Testing Terminology Anomaly Bug Exception
Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents (IEEE). Bug Things the software does that it is not supposed to do Something the software doesn't do that it is supposed to. Exception An event that causes suspension of normal program execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).
20
Testing Terminology: Testing vs. Debugging
Evaluating software to determine conformance to some objective. Debugging Finding a fault, given a failure.
21
Testing Terminology: The RIP Fault/Failure Model
Three conditions are necessary for failures to be observable. Reachability: The location of the fault must be reached. Infection: After executing the location, the state of the program must be incorrect. Propagation: The infected state must propagate to cause incorrect program output.
22
Testing Terminology Validation Verification
Does software meet its goals? Are we building the right product? User-centric! Verification Are we building the product right? E.g., does the code reflect the design? Developer-centric! Name for the artifact being tested Implementation under test (IUT) Method under test (MUT), object under test (OUT) Class/component under test (CUT), etc.
23
Testing Terminology: Test Plan
A test plan specifies how we will demonstrate that the software is free of faults and behaves according to the requirements specification A test plan breaks the testing process into specific tests, addressing specific data items and values Each test has a test specification that documents the purpose of the test If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results
24
Testing Terminology: Test Oracle
A test oracle is the set of predicted results for a set of tests, and is used to determine the success of testing Test oracles are extremely difficult to create and are ideally created from the requirements specification
25
Testing Terminology: Test Case
A test case is a specification of The pretest state of the IUT Prefix values: values needed to put software into state required before test execution. A set of inputs to the system Postfix values: values that need to be sent to the software after test execution. The expected results
26
Testing Terminology Test Point Test suite Domain
A specific value for test case input and state variables. Domain Set of values that input or state variables of IUT may take. Heuristics for test point selection Equivalence classes – one values in set represents all values in set. Partition testing techniques Boundary value analysis Special values testing Test suite A collection of test cases.
27
Testing Terminology Fault model Test strategy Test model Test design
A description about where faults are likely to occur in a program. Test strategy An algorithm or heuristic for creating test cases from a representation, an implementation or a test model Test model A description of relationships between elements of a representation or implementation. Test design The process of creating a test suite using a test strategy. Test effectiveness: relative ability of testing strategy to find bugs. Test efficiency: relative cost of finding bugs.
28
Testing Terminology: Testing Strategies
Responsibility-based test design Behavioural, functional, black-box, etc. testing Implementation-based test design Relies on source code,, e.g., white-box Fault-based testing
29
A Classification of Testing: The V Model
Requirements Analysis Acceptance Test Architectural Design System Test Subsystem Design Integration Test Detailed Design Module Test Implementation Unit Test
30
A Classification of Testing
Graphs Logical expressions Input domain characterizations Syntactic descriptions
31
Test-Case Exercise UML class model for Figure Hierarchy
32
The Polygon Class abstract class Polygon {
abstract void draw(int r, int g, int b); /* color closed area*/ abstract void erase(); /* set to background rgb */ abstract float area(); /* return area */ abstract float perimeter(); /* return sum of sides */ abstract Point center(); /* return centroid pixel */ }
33
The Triangle Class class Triangle extends Polygon {
public Triangle(LineSegment a, LineSegment b, LineSegment c){…} public void setA(LineSegment a) {…} public void setB(LineSegment b) {…} public void setC(LineSegment c) {…} public LineSegment getA() {…} public LineSegment getB() {…} public LineSegment getC() {…} public boolean is_isoceles(){…} public boolean is_scalene(){…} public boolean is_equilateral(){…} public void draw(int r, int g, int b){…} public void erase(){…} public float area(){…} public float perimeter(){…} public Point center(){…} }
34
The LineSegment Class class LineSegment extends Figure {
public LineSegment(Point x1, Point y1, Point x2, Point y2){…} public void setx1(Point x1) {…} public void sety1(Point y1) {…} public void setx2(Point x2) {…} public void sety2(Point y2) {…} public Point getx1() {…} public Point gety1() {…} public Point getx2() {…} public Point gety2() {…} }
35
Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. The class is used to determine if a triangle is: isosceles, equilateral or scalene. A triangle is isosceles if two of its sides are equal. A triangle is equilateral if all its sides are equal. A triangle is scalene if its sides are of different sizes.
36
Test-Case Exercise A valid triangle must meet two conditions:
No side may have a length of zero. Each side must be smaller than the sum of all sides divided by two. i.e. Given s = (a + b + c)/2, then a < s, b < s, c < s must hold. Stated differently, a < s && b < s && c < s a < b + c, b < a + c and c < a + b.
37
Test-Case Exercise Specify as many test cases as you can for the Triangle class defined above. Test Case Description Test Case Input a b c Expected Output 1 Valid isosceles triangle 3 4 Isosceles 2 28 29 30 31 32 33 34 35
38
Solution To Test Case Exercise
39
Solution To Test Case Exercise
40
Solution To Test Case Exercise
41
Solution To Test Case Exercise
42
The C++ IntSet Class Test Case Design Exercise 2
Design test cases to test the methods (ignore the constructor and destructor) of the IntSet class. The class performs operations on a single set – duplications are not allowed If add(val) is called and val is already in the set, then the Duplicate exception is thrown. class IntSet { public: //operations on a set of integers IntSet(); /* constructor */ ~IntSet(); /* destructor */ IntSet& add (int); /* add a member */ IntSet& remove (int); /* remove a member */ IntSet& clear (int); /* remove all members */ int isMember (int arg); /* Is arg a member */ int extent (); /* Number of elements */ int isEmpty (); /* Empty or not */ } Design test cases to test the methods (ignore the constructor and destructor).
43
Who Tests the Software? developer independent tester
Understands the system but, will test "gently“ and, is driven by "delivery" Must learn about the system but, will attempt to break it and, is driven by quality
44
Testing Strategy We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’ For conventional software The module (component) is our initial focus Integration of modules follows For OO software “testing in the small” OO class Has attributes and operations and implies communication and collaboration
45
Junit Overview
46
Unit Testing module to be tested results software engineer test cases
interface local data structures boundary conditions independent paths error handling paths The module being tested should be reviewed in context of the requirements specification The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax A set of test cases is formulated and input and the results are evaluated
47
Unit Test Environment test cases RESULTS driver Module stub stub
interface local data structures Module boundary conditions independent paths error handling paths stub stub test cases RESULTS
48
What is JUnit? JUnit is an open source unit testing framework for Java created by Kent Beck and Erich Gamma. JUnit features include: Assertions for testing expected results Test fixtures for sharing common test data Test suites for easily organizing and running tests Graphical and textual test runners
49
JUnit Terminology Fixture: a set of objects against which tests are run A test fixture is ideal when two or more tests are to be executed for a common set of objects. In these scenarios, a test fixture avoids duplicating the initialization tasks before each test and the cleanup activities after each test setup: a method which sets up the fixture, called before each test is executed. teardown: a method to tear down the fixture, called after each test is executed. Test Case: a class which defines the fixture to run multiple tests - create a subclass of TestCase - add an instance variable for each part of the fixture - override setUp() to initialize the variables and allocate resource - override tearDown() to release any permanent storage, etc. Test Suite: a collection of test cases.
50
The JUnit Framework
51
JUnit 3
52
How To Write JUnit 3 Test Cases
Import the Junit framework: import junit.framework.*; Extend the “TestCase” class extend junit.framework.TestCase; Make use of special method-name conventions: Begin each test method name with the prefix ‘test’. Each test case is written in a method that has a name that begins with “test” Create a mehod with the name setUp to initialize your object, etc. Create a mehod with the name tearDown to clean-up after your test. etc.
53
A Case Study class Money { private int fAmount;
private String fCurrency; public Money(int amount, String currency) { fAmount= amount; fCurrency= currency; } public int amount() { return fAmount; } public String currency() { return fCurrency; } public Money add(Money m) { return new Money(amount()+m.amount(), currency()); } }
54
How to Write A Simple Standalone TestCase
import junit.framework.*; public class MoneyTest extends TestCase { //… public void testSimpleAdd() { Money m12 = new Money(12, "Dollar"); // (1) Money m14 = new Money(14, "Dollar"); Money expected = new Money(26, "Dollar"); Money result = m12.add(m14); // (2) Assert.assertTrue(expected.equals(result)); // (3) } Creates the objects we will interact with during the test. This testing context is commonly referred to as a test's fixture. All the fixture contains for the testSimpleAdd test are some Money objects. Exercises the objects in the fixture. Verifies the result
55
Assert assertEquals(expected, actual)
assertEquals(message, expected, actual) assertEquals(expected, actual, delta) assertEquals(message, expected, actual, delta) assertFalse(condition) assertFalse(message, condition) AssertNotNull(object) AssertNotNull(message, object) AssertNotSame(expected, actual) AssertNotSame(message, expected, actual) assertTrue(condition) assertTrue(message, condition)
56
Start to Use it 2. Installation unzip the junit.zip file
1. Download the latest version of JUnit from 2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type: java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type: java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type: java junit.swingui.TestRunner junit.samples.AllTests Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path Important: Don't install the junit.jar into the extension directory of your JDK installation. If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse
57
A Second Example import junit.framework.*;
public interface ISorting { //etc. } public SortingComparison implements ISorting { //... } public void bubbleSort(int[] a); //implements bubblesort algorithm import junit.framework.*; public class SortingTest extends TestCase { SortingComparison obj;//instance variable in the fixture public SortingTest(String name) { super(name); } public void testBubble() { //SortingComparison obj = new SortingComparison(); int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0}; int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000}; obj.bubbleSort(qArray); obj.printArray(qArray); obj.printArray(rArray); assertTrue(java.util.Arrays.equals(qArray, rArray)); } public void setUp() { obj = new SortingComparison(); } public void testSimpleTest() { int answer = 2; assertEquals((1+1), answer); } public void tearDown () { //do something }
58
Example code 1 import junit.framework.TestCase;
public class JunitExamples extends TestCase { SortingComparison obj;//instance variable in the fixture public JunitExamples(String name) { super(name); } public void setUp() { obj = new SortingComparison(); public void tearDown () { //do something public void testBubble(){ //SortingComparison obj = new SortingComparison(); int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0}; int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000}; obj.bubbleSort(qArray); obj.printArray(qArray); obj.printArray(rArray); assertTrue(java.util.Arrays.equals(qArray, rArray)); }//end of method }//end of class
59
Example code 2 import junit.framework.TestCase;
public class JunitExamples extends TestCase { SortingComparison obj;//instance variable in the fixture public JunitExamples(String name) { super(name); } public void setUp() { obj = new SortingComparison(); public void tearDown () { //do something public void testIsosceles (){ int side1 =3, side2 = 3, side3 = 0; boolean result = is_isosceles(side1, side2, side3); assertTrue(result == true); }//end of method}//end of class
60
Running JUnit java junit.textui.TestRunner JavaClassfileName
java junit.swingui.TestRunner JavaClassfileName java junit.awtui.TestRunner JavaClassfileName Use eclipse Download the JUnit and integrate in eclipse Create your java project and the java application to be tested. then … (next slide)
61
This will be filled in automatically
JUnit in Eclipse To create a test class, select File New Other... Java, JUnit, TestCase and enter the name of the class you will test Fill this in This will be filled in automatically
62
Running JUnit Second, use this pulldown menu
First, select a Test class Third, Run As JUnit Test
63
Results Your results are here
64
Start to Use it 2. Installation unzip the junit.zip file
1. Download the latest version of JUnit from 2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type: java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type: java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type: java junit.swingui.TestRunner junit.samples.AllTests Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path Important: Don't install the junit.jar into the extension directory of your JDK installation. If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse
65
JUnit 4
66
Comparing JUnit 3 to JUnit 4
All the old assertXXX methods are the same Most things are about equally easy JUnit 4 makes it easier to test that exceptions are thrown when they should be JUnit 4 can still run JUnit 3 tests JUnit 4 provides protection against infinite loops JUnit 4 has some additional features
67
Migrating to JUnit 4 JUnit 4 requires Java 5 or newer
Don’t extend junit.framework.TestCase; just use an ordinary class Import org.junit.* and org.junit.Assert.* Use a static import for org.junit.Assert.* Static imports replace inheritance from junit.framework.TestCase Use annotations instead of special method names: Instead of a setUp method, before some method Instead of a tearDown method, before some method Instead of beginning test method names with ‘test’, before each test method
68
Testing Strategies
69
Testing Strategy unit test integration test Black Box Testing
White Box Testing system test validation test
70
Unit Testing module to be tested
software engineer results module to be tested test cases interface local data structures boundary conditions independent paths error handling paths The module being tested should be reviewed in context of the requirements specification The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax A set of test cases is formulated and input and the results are evaluated
71
Unit Test Environment test cases RESULTS driver Module stub stub
interface local data structures Module boundary conditions independent paths error handling paths stub stub test cases RESULTS
72
Integration Testing The goal is to ensure that groups of components work together as specified in the requirements document Four kinds of integration tests exist Structure tests Functional tests Stress tests Performance tests
73
Integration Testing Strategies
Options: • the “big bang” approach • an incremental construction strategy
74
Top Down Integration A top module is tested with stubs B F G
stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E
75
Bottom-Up Integration
F G drivers are replaced one at a time, "depth first" C worker modules are grouped into builds and integrated D E cluster
76
Sandwich Testing cluster A Top modules are tested with stubs B F G C
Worker modules are grouped into builds and integrated D E cluster
77
System Testing The goal is to ensure that all elements that interact to solve the problem do so correctly. Software, databases, hardware, people, procedures. Overall functionality and performance must be achieved.
78
Validation Testing The goal is to ensure that the system actually does what the customer expects it to do – fulfils requirements Testing is carried out by customers mimicking real world activities Customers should also intentionally enter erroneous values to determine the system behavior in those instances
79
Black Box Testing Testing the functionality of the program
The tester either knows nothing about the internal structure of the code or otherwise ignores this information Test cases are formulated based on expected output of methods Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behaviour is the same
80
White Box Testing The tester uses knowledge of the programming constructs to determine the test cases to use If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations Similarly, conditions would be tested for true and false
81
High Order Testing Validation testing System testing
Focus is on software requirements System testing Focus is on system integration Alpha/Beta testing Focus is on customer usage Recovery testing forces the software to fail in a variety of ways and verifies that recovery is properly performed Security testing verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration Stress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volume Performance Testing test the run-time performance of software within the context of an integrated system
82
Strategic Issues State testing objectives explicitly.
Understand the users of the software and develop a profile for each user category. Develop a testing plan Build “robust” software that is designed to test itself Use effective formal technical reviews as a filter prior to testing Conduct formal technical reviews to assess the test strategy and test cases themselves. Develop a continuous improvement approach for the testing process.
83
Testing Steps Determine what the test is supposed to measure
Decide how to carry out the tests Develop the test cases Determine the expected results of each test (test oracle) Execute the tests Compare results to the test oracle
84
Object-Oriented Testing
begins by evaluating the correctness and consistency of the OOA and OOD models testing strategy changes the concept of the ‘unit’ broadens due to encapsulation integration focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario validation uses conventional black box methods test case design draws on conventional methods, but also encompasses special features
85
Broadening the View of “Testing”
“It can be argued that the review of OO analysis and design models is especially useful because the same semantic constructs (e.g., classes, attributes, operations, messages) appear at the analysis, design, and code level. Therefore, a problem in the definition of class attributes that is uncovered during analysis will circumvent side effects that might occur if the problem were not discovered until design or code (or even the next iteration of analysis)” - Pressman.
86
OOT Strategy class testing is the equivalent of unit testing
operations within the class are tested the state behavior of the class is examined integration applied three different strategies thread-based testing—integrates the set of classes required to respond to one input or event use-based testing—integrates the set of classes required to respond to one use case cluster testing—integrates the set of classes required to demonstrate one collaboration
87
Smoke Testing A common approach for creating “daily builds” for product software Smoke testing steps: Software components that have been translated into code are integrated into a “build.” A build includes all data files, libraries, reusable modules, and engineered components that are required to implement one or more product functions. A series of tests is designed to expose errors that will keep the build from properly performing its function. The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind schedule. The build is integrated with other builds and the entire product (in its current form) is smoke tested daily. The integration approach may be top down or bottom up.
88
Debugging: A Diagnostic Process
89
The Debugging Process Debugging test cases results new test cases
regression tests suspected causes corrections Debugging identified causes
90
Debugging Effort time required to diagnose the symptom and
determine the cause time required to correct the error and conduct regression tests
91
Symptoms & Causes symptom cause symptom and cause may be
geographically separated symptom may disappear when another problem is fixed cause may be due to a combination of non-errors cause may be due to a system or compiler error cause may be due to assumptions that everyone believes symptom cause symptom may be intermittent
92
Consequences of Bugs infectious damage catastrophic extreme serious
disturbing annoying mild Bug Type Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.
93
Debugging Techniques brute force / testing backtracking
Cause-elimination Induction (hypothesis cause) deduction (cause hypothesis)
94
Debugging: Final Thoughts
symptom you're seeing. Don't run off half-baked, think about the 1. more insight. Use tools (e.g., dynamic debugger) to gain 2. If at an impasse, get help from someone else. 3. Be absolutely sure to conduct regression tests when you do "fix" the bug. 4.
95
White-Box Testing & Control Flow Graphs
96
Testing Strategy White Box Testing unit test integration test
Black Box Testing system test validation test
97
White-box testing: control-flow graph
Directed graph? A control flow graph of a program is a graph that shows the paths that may be traversed when the program is executed. Vertices are statements in the program
98
Control-flow Graph int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else
3 4 5 6 7 8 int n = 0; do{ if(n++ % 2 == 1) dBase[i]++; else if(dBase[i] > 100) dBase[i] -= n; dBase[i] += n; }while(n < 20); 1 2 4 3 5 6 7
99
Cyclomatic Complexity – V(G)
If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by: V(G) = e – n + 2 Or V(G) = the number of decision nodes + 1
100
Cyclomatic Complexity – V(G)
1 2 3 4 5 6 7 8 cyclomatic complexity V(G) = e – n + 2 = 11 – 7 = 4 Or V(G) = the number of decision nodes + 1 = = 4
101
Cyclomatic Complexity
A number of industry studies have indicated that the higher V(G), the higher the probability or errors. modules V(G) modules in this range are more error prone
102
Cyclomatic Complexity – V(G)
1 2 3 4 5 6 7 8 The cyclomatic complexity is a measure of the number of linearly independent paths in the graph. An independent path must move along at least one edge not previously traversed What are the independent paths for this graph? …8
103
White-box testing: Code coverage
White box testing methods emphasize code coverage Statement coverage Every statement is executed at least once Branch coverage Every branch is exercised at least once i.e. every path from a node is traversed at least once Once true and once false evaluation Multiple condition coverage All combinations of simple conditions are exercised Basis-path model Exercise V(G) independent paths Data flow coverage Evaluation of state of instance variables.
104
Statement Coverage 1 2 3 4 5 6 7 8 Write enough test cases to execute each statement al least once, May miss bugs for complex predicates and loops
105
Statement Coverage int num = 0; do { }while(num != getValue(num));
1 2 3 4 5 6 7 8 int num = 0; do { if(num > 5 && (sum-num)<findMax(num)) { if(transform(num)) transpose(Math.pow(num, 2)) else sum += num + 5; } sum += findMax(num); }while(num != getValue(num)); 1 2 3 6 5 4 7
106
Branch Coverage Determine number of branches
1 2 3 4 5 6 7 8 Determine number of branches Compute path for each branch Write test cases to exercise each path
107
Multiple Condition Coverage
1 2 3 4 5 6 7 8 Determine true/false combinations for conditions Write test cases to exercise each combination The required test cases may be determined using the truth table for the logical operator of the condition.
108
Multiple Condition Coverage
if(num > 5 || isValid(num) && (sum-num<findMax(num) || sum = 100) && isDivisible(num, 7) Test Case # num > 5 isValid(num) (sum-num) < findMax(num) sum=100 isDiv (num, 7) 1 F 2 T 3 4 5 6 7 8 9 10 11
109
Basis-path Testing Compute the cyclomatic complexity
1 2 3 4 5 6 7 8 Compute the cyclomatic complexity Derive independent paths …7-8 Write test cases to exercise each path
110
Example 1 V(G) = 2 + 1 = 3 Statement coverage requires 2 test cases
Branch coverage requires 2 test cases Basis-path coverage requires 3 test cases There are four distinct entry-exit paths Which should be chosen? 1 2 4 3 5 6 7
111
Example 2 A D What is V(G)? How many test cases are required for statement coverage? How many test cases are required for branch coverage? How many test cases are required for basis-path coverage? How many unique entry-exit paths exist? LIST THE PATHS F E K L
112
Introduction to Mutation Testing & Program Perturbation
113
What is Mutation Testing?
Mutation Testing is a testing technique that focuses on measuring the adequacy of test cases. Mutation Testing is NOT a testing strategy like path or data-flow testing. It does not outline test data selection criteria. Mutation Testing should be used in conjunction with traditional testing techniques, not instead of them.
114
Mutation Testing Faults are introduced into a program by creating many versions of the program Each version is called a mutant. Each mutant contains a single fault. Test cases are applied to the original program and to the mutant program. The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.
115
Example of a Program Mutation
1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x < y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 }
116
Mutation Testing Process
Use mutation operators to introduce errors into the source program One error per mutant Validate the test suite with unmodified program (insures the test suite functions properly) Run test suite on mutant programs Analyze result to determine how many non-equivalent mutants were killed Fix test suite to kill all live mutants until Mutation Score = 1.0 or specified a threshold score is reached e.g. 0.96
117
Mutation Testing Process
Process (cont.) Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3
118
Test Case Adequacy A test case is adequate if it is useful in detecting faults in a program. A test case can be shown to be adequate by finding at least one mutant program that generates a different output than does the original program for that test case. If the original program and all mutant programs generate the same output, the test case is inadequate.
119
Terminology Mutation testing Mutation
The creation of a set of mutant programs of the program being tested. Mutation a single syntactic change that is made to a program statement. Each mutant differs from the original program by one mutation.
120
Terminology Mutant Mutant Killed Mutation Score Mutation Operator
A faulty program; the original with one fault introduced Mutant Killed Test suite detects the mutant Mutation Score #of killed mutants/ # of non-equivalent mutants Mutation Operator The function that changes the source code to create a mutant
121
Mutation Operators Replace each operand with every other syntactically correct operand Modify expressions by replacing operators and inserting new operators Delete entire statements
122
Categories of Mutation Operators
Operand Replacement Operators: Replace a single operand with another operand or constant. E.g., if(x > y) if (5 > y) Replacing x by constant 5. if (x > 5) Replacing y by constant 5. if (y > x) Replacing x and y with each other. E.g., if all operators are {+,-,*,**,/} then the following expression a = b * (c - d) will generate 8 mutants: 4 by replacing * 4 by replacing -.
123
Categories of Mutation Operators
Expression Modification Operators: Replace an operator or insert new operators. E.g., if (x == y) if (x >= y) Replacing == by >=. if (x == ++y) Inserting ++.
124
Categories of Mutation Operators
Statement Modification Operators: E.g., Delete the else part of the if-else statement. Delete the entire if-else statement. Replace line 3 by a return statement. 1 int max(int x, int y) 2 { 3 int mx = x; 4 if (x > y) 5 mx = x; 6 else 7 mx = y; 8 return mx; 9 } return x;
125
Mutation System Example
The Mothra mutation system for FORTRAN77 supports 22 mutation operators. E.g., absolute value insertion constant for array reference replacement GOTO label replacement RETURN statement replacement statement deletion unary operator insertion logical connector replacement comparable array name replacement
126
Why Does Mutation Testing Work?
The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.
127
The Competent Programmer Hypothesis
Programmers are generally very competent and do not create “random” programs. For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program. An incorrect program can be created from a correct program by making some minor change to the correct program.
128
Mutation Testing Algorithm
Generate program test cases. Run each test case against the original program. If the output is incorrect, the program must be modified and re-tested. If the output is correct go to the next step ... Construct mutants E.g. using a tool like Mothra.
129
Mutation Testing Algorithm (Cont’d)
Execute each test case against each alive mutant. If the output of the mutant differs from the output of the original program, the mutant is considered incorrect and is killed. Two kinds of mutants survive: Functionally equivalent to the original program: Cannot be killed. Killable: Test cases are insufficient to kill the mutant. New test cases must be created.
130
Mutation Score The mutation score for a set of test cases is the percentage of non-equivalent mutants killed by the test data. Mutation Score = 100 * D / (N - E) D = Dead mutants N = Number of mutants E = Number of equivalent mutants A set of test cases is mutation adequate if its mutation score is 100%.
131
Evaluation Theoretical and experimental results have shown that mutation testing is an effective approach to measuring the adequacy of test cases. The major drawback of mutation testing is the cost of generating the mutants and executing each test case against them. Very expensive to run (time, processor intensive, memory)
132
Program Perturbation Program Perturbation is a technique to test a program’s robustness. It is based on unexpectedly changing the values of program data during run-time.
133
Software Failure Hypothesis
Program perturbation is based on the three part software failure hypothesis: Execution: The fault must be executed. Infection: The fault must change the data state of the computation directly after the fault location. Propagation: The erroneous data state must propagate to an output variable.
134
Program Perturbation Process
The tester must: inject faults in the data state of an executing program; trace the impact of the injected fault on the program’s output. The injection is performed by applying a perturbation function that changes the program’s data state.
135
The Perturbation Function
The perturbation function is a mathematical function that: takes a data state as its input; changes the data state according to some specified criteria; produces a modified data state as output.
136
The Fault Injection A program location N is chosen along with a set of input variables I that are in scope at location N. The program is executed until location N. When execution arrives at location N, the resulting data state is changed (perturbed). The subsequent execution will either fail or succeed.
137
Program Perturbation Example
Assume the following perturbation function: 1. int perturbation (int x) 2. { 3. int newX; 4. newX = x + 20; 5. return newX; 6. }
138
Example of a Fault Injection
1. main() 2. { 3. int x; 4. x = ReadInt(); 4.1 x = perturbation(x); 5. if (x > 0) printf(“X positive”); 7. else printf(“X negative”); 9. } 1. main() 2. { 3. int x; 4. x = ReadInt(); 5. if (x > 0) 6. printf(“X positive”); 7. else 8. printf(“X negative”); 9. }
139
What Perturbation Testing is and is Not
Perturbation testing is NOT a testing technique that outlines test selection and coverage criteria. Rather, perturbation testing is a technique that can be used to measure the reliability of the software (tolerance to faults).
140
Evaluation The program is repeatedly executed and injected with faults during each execution. The ratio of the number of failures detected divided by the total number of executions is used to predict failure tolerance.
141
Summary & The End Qu es ti ons? ______________________
Devon M. Simmonds Computer Science Department University of North Carolina Wilmington ____________________________________________ _________________
142
Why is testing so difficult?
1 3 2 5 4 loop < =20 X a b c d e There are about 1014 possible paths How long would it take to execute all paths? About 3,170 one test/millisecond
143
Example 2 A What is V(G)? - 7 How many test cases are required for statement coverage? - 4 How many test cases are required for branch coverage? - 4 How many test cases are required for basis-path coverage? - 7 How many unique entry-exit paths exist? – 16 D F E K L ABDEGKMQS ABDEGKNQS ABDEHKMQS ABDEHKNQS ACDEGKMQS ACDEGKNQS ACDEHKMQS ACDEHKNQS ACDFJLPRS ACDFJLQRS ACDFILPRS ACDFILQRS ABDFJLPRS ABDFJLQRS ABDFILPRS ABDFILQRS
144
Example 2 A D F E K L ABDEGKMQS ABDEGKNQS ABDEHKMQS ABDEHKNQS
ACDEGKMQS ACDEGKNQS ACDEHKMQS ACDEHKNQS F E K L ACDFJLPRS ACDFJLQRS ACDFILPRS ACDFILQRS ABDFJLPRS ABDFJLQRS ABDFILPRS ABDFILQRS
145
Structure setUp() Storing the fixture's objects in instance variables of your TestCase subclass and initialize them by overriding the setUp method tearDown() Releasing the fixture’s run() Defining how to run an individual test case. Defining how to run a test suite. testCase()
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.