Topic 12Summer ICS 52: Introduction to Software Engineering Lecture Notes for Summer Quarter, 2003 Michele Rousseau Topic 12 Partially based on lecture notes written by Sommerville, Frost, Van Der Hoek, Taylor & Tonne. Duplication of course material for any commercial purpose without the written permission of the lecturers is prohibited

Topic 12Summer Verification and Validation Informal Requirements Formal Specification Software Implementation Validation Verification Verification: is implementation consistent with requirements specification? Validation: does the system meet the customer’s/user’s needs?

Topic 12Summer Software Quality: assessment by V&V l Software process must include verification and validation to measure product qualities correctness, reliability, robustness efficiency, usability, understandability verifiability, maintainability reusability, portability, interoperability, real-time, safety, security, survivability, accuracy l Products can be improved by improving the process by which they are developed and assessed

Topic 12Summer Testing Terminology l Failure: Incorrect or unexpected output, based on specifications Symptom of a fault l Fault: Invalid execution state Symptom of an error May or may not produce a failure l Error: Defect or anomaly or “bug” in source code May or may not produce a fault

Topic 12Summer Examples: Faults, Errors, and Failures l Suppose node 6 should be X:= C*(A+2*B) Failure/Fault-less error: - Suppose the inputs are (A=2,B=1) – the executed path will be (1,2,4,5,7,8) which will not reveal this fault because 6 is not executed - Suppose the inputs are (A=-2,B=-1) – the executed path will be (1,2,3,5,6,8) which will not reveal the fault because C = 0 l Need to make sure proper test cases are selected the definitions of C at nodes 3 and 4 both affect the use of C at node 6 - executing path (1,2,4,5,6,8) will reveal the failure, but only if B != 0 - (e.g. Inputs (A=1,B=-2) )

Topic 12Summer Software Testing l Exercising a system [component] on some predetermined input data capturing the behavior and output data comparing with test oracle for the purposes of »identifying inconsistencies »verifying consistency between actual results and specification  to provide confidence in consistency with requirements and measurable qualities  to demonstrate subjective qualities »validating against user needs l Limitations only as good as the test data selected subject to capabilities of test oracle

Topic 12Summer Definition to execution (Levels of testing) l System Testing Defined at Requirements -> Run after integration testing l Integration Testing Defined at Design -> Run after Module Testing l Module Testing Defined at Design -> Run after Unit Testing l Unit Testing Defined at Implementation -> Run after Implementation of each unit l Regression Testing (testing after Change) Defined throughout the process -> Run after modifcations

Topic 12Summer Testing-oriented Life Cycle Process

Topic 12Summer How to make the most of limited resources? Fundamental Testing Questions l Test Criteria: What should we test? l Test Oracle: Is the test correct? l Test Adequacy: How much is enough? l Test Process: Is our testing effective?

Topic 12Summer Practical Issues l Purpose of testing Fault detection High assurance of reliability Performance/stress/load Regression testing of new versions l Conflicting considerations safety, liability, risk, customer satisfaction, resources, schedule, market windows and share l Test Selection is a sampling technique choose a finite set from an infinite domain

Topic 12Summer Test Criteria l Testing must select a subset of test cases that are likely to reveal failures l Test Criteria provide the guidelines, rules, strategy by which test cases are selected actual test data conditions on test data requirements on test data l Equivalence partitioning is the typical approach a test of any value in a given class is equivalent to a test of any other value in that class if a test case in a class reveals a failure, then any other test case in that class should reveal the failure some approaches limit conclusions to some chosen class of errors and/or failures

Topic 12Summer Test Adequacy l Coverage metrics when sufficient percentage of the program structure has been exercised l Empirical assurance when failures/test curve flatten out l Error seeding percentage of seeded faults found is proportional to the percentage of real faults found l Independent testing faults found in common are representative of total population of faults

Topic 12Summer Functional and Structural Testing l Functional Testing Test cases selected based on specification Views program/component as black box l Structural Testing Test cases selected based on structure of code Views program /component as white box (also called glass box testing)

Topic 12Summer Black Box vs. White Box Testing

Topic 12Summer Structural (White Box) Testing l Use source code to derive test cases l Build flow graph of program l State test requirements in terms of graph coverage l Various types of coverage identified l Choose test cases that guarantee different types of coverage All-Statements coverage All-Branches coverage All-Edges All-Paths

Topic 12Summer Graph representation of control flow and data flow relationships Flow Graphs l Control Flow The partial order of statement execution, as defined by the semantics of the language l Data Flow The flow of values from definitions of a variable to its uses

Topic 12Summer function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X > 10) loop X := X – 10; exit when X = 10; end loop; if (Y < 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y – 20; end if; return 2*X + Y; end P; A Sample Program to Test

Topic 12Summer ,3,45 6 9´ TT F F 9 T F 7 T F 9a9a 9b9b P’s Control Flow Graph (CFG) function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X > 10) loop X := X – 10; exit when X = 10; end loop; if (Y < 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y – 20; end if; return 2*X + Y; end P;

Topic 12Summer All-Statements Coverage l Select test cases such that every node in the graph is visited Also called node coverage »Guarantees that every statement in the source code is executed at least once l Selects minimal number of test cases

Topic 12Summer ,3, T F 9 T F 7 T F At least 2 test cases needed Example all-statements-adequate test set: (X = 20, Y = 10) (X = 20, Y = 30) All-Statements Coverage of P

Topic 12Summer All-Branches Coverage l Select test cases such that every branch in the graph is visited »Guarantees that every branch in the source code is executed at least once l More thorough than All-Statements coverage More likely to reveal logical errors

Topic 12Summer ,3, T F 9 T F 7 T F At least 2 test cases needed Example all-branches-adequate test set: (X = 20, Y = 10) (X = 15, Y = 30) All-Branches Coverage of P

Topic 12Summer All-Edges Coverage l Select test cases such that every edge in the graph is visited »Takes complex statements into consideration – treats them as separate nodes l More thorough than All-Branches coverage More likely to reveal logical errors

Topic 12Summer ,3,45 6 9b9b TT F F 9a9a T F 7 T F At least 3 test cases needed Example all-edges-adequate test set: (X = 20, Y = 10) (X = 5, Y = 30) (X = 21, Y = 10) All-Edges Coverage of P

Topic 12Summer All-Paths Coverage l Path coverage Select test cases such that every path in the graph is visited Loops are a problem »0, 1, average, max iterations l Most thorough… l …but is it feasible?

Topic 12Summer ,3,45 6 9b9b TT F F 9a9a T F 7 T F Infinitely many test cases needed Example all-paths-adequate test set: (X = 5, Y = 10) (X = 15, Y = 10) (X = 25, Y = 10) (X = 35, Y = 10) … All-Paths Coverage of P

Topic 12Summer ,3,45 6 9b9b T F 9a9a T F Y X X Y YX X Y Y X X X T F 7 T F X X P’s Control and Data Flow Graph

Topic 12Summer Subsumption of Criteria l C1 subsumes C2 if any C1-adequate testT is also C2-adequate But some T1 satisfying C1 may detect fewer faults than some T2 satisfying C2

Topic 12Summer all-statements all-edges boundary-interior loop testing min-max loop testing all-paths all-defs all-uses all-DU-paths all-p-usesall-c-uses C1C1 C2C2 subsumes all-branches Structural Subsumption Hierarchy

Topic 12Summer Equivalence partitioning l Identify the set of all inputs l Identify possible bases for subdividing the input Domain: size, order, structure correctness stated requirements & your smarts l Divide input set into (sub)domains using the basis “Equivalence partitions” Subdomains may overlap (may not be a partition) l Select representative test cases from each subdomain one test case may suffice, more is better

Topic 12Summer Example 1float homeworkAverage(float[] scores) { 2 float min = 99999; 3 float total = 0; 4 for (int i = 0 ; i < scores.length ; i++) { 5 if (scores[i] < min) 6 min = scores[i]; 7 total += scores[i]; 8 } 9 total = total – min; 10 return total / (scores.length – 1); 11}

Topic 12Summer Possible Bases l Array length empty array one element 2 or 3 elements 4 or more elements Input domain: float[] Basis: array length one small empty large } subdomains, eq. partitions

Topic 12Summer Possible Bases l Position of minimum score Smallest element first Smallest element in middle Smallest element last Input domain: float[] Basis: position of minima somewhere in middle first last

Topic 12Summer Possible Bases l Number of minima Unique minimum A few minima All minima Input domain: float[] Basis: number of minima all data equal 1 minimum 2 minima

Topic 12Summer Equivalence partitions & white-box testing The basis - equivalence partition - test case approach works for structural testing, too. A basis could be nodes, edges, or paths. Each node, edge, or path defines a partition of the input.

Topic 12Summer Challenges l Structural testing can cover all nodes or edges without revealing obvious faults No matter what input, program always returns 0 l Some nodes, edges, or loop combinations may be infeasible Unreachable/unexecutable code l “Thoroughness” A test suite that guarantees edge coverage also guarantees node coverage… …but it may not find as many faults as a different test suite that only guarantees node coverage

Topic 12Summer Limitations of Structural Testing l Interactive programs l Listeners, event-driven programs l Concurrent programs or tasks; threads l Exceptions l Self-modifying programs (assembly language) l Objects loaded over the network l Super-class constructors l Asynchronous garbage collection

Topic 12Summer Summary of Structural Testing l Conceptually simple l Relies on access to source l Not applicable in all situations l Acts in ignorance of specifications

Topic 12Summer Ensure that changes made during maintenance do not destroy existing functionality Regression Testing l Permanent suite of test cases Saves effort creating test cases Provides record of existing functionality l Add new test cases and delete obsolete ones when necessary

Topic 12Summer Cost reduction: Select minimal subset of regression test suite that tests the changes Selective Regression Testing l Analyze relationship between the test cases and the software entities they cover l Use information about changes to select test cases for new version

Topic 12Summer = modified entity System Under Test Test 2Test 1Test 3 = exercises relation = entity Simple Example

Topic 12Summer System Under Test Test 3 = exercises relation = entity = modified entity Selective Regression Testing

Topic 12Summer Cost of running necessary tests Retest All Cost-Ineffective Selective Retest Cost of running necessary tests Cost Cost-Effective Selective Retest Cost of running unnecessary tests Cost of additional analysis The Cost-Effectiveness Tradeoff

Topic 12Summer The Reality of Software Testing l One of the most widely-used, practical class of techniques for revealing faults in software l An area of software engineering showing the greatest gap between theory and practice