1 Software Testing & Quality Assurance Lecture 10 Created by: Paulo Alencar Modified by: Frank Xu

2 Overview  Structural Testing  Introduction – General Concepts  Flow Graph Testing  DD-Paths  Test Coverage Metrics  Basis Path Testing  Guidelines and Observations  Data Flow Testing  Hybrid Methods  Retrospective on Structural Testing

3 Path Testing Path Testing is focusing on test techniques that are based on the selection of test paths though a program graph. If the set of paths is properly chosen, then we can claim that we have achieved a measure of test thoroughness. The fault assumption for path testing techniques is that something has gone wrong with the software that makes it take a different path than the one intended. Structurally, a path is a sequence of statements in a program unit. Semantically, a path is an execution instance of the program unit. For software testing we are interested in entry-exit paths.

4 Path Testing Process Input: –Source code and a path selection criterion Process: –Generation of a CFG –Selection of Paths –Generation of Test Input Data –Feasibility Test of a Path –Evaluation of Program’s Output for the Selected Test Cases

5 Flow Graphs – Use in determining Paths for Testing - Revisited

6 Program Graph to DD-Path Graph first E M J K H C B I F G last D L N O

7 Topological Paths We can identify eight possible paths through the previous diagram 1. first – A – B – C – E – F – G – O – last 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 4. first – A – B – D – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 6. first – A – B – D – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last 8. first – A – B – D – E – F – H – J – L – M – N – O – last

8 Feasible Paths If we assume that the logic of the program dictates that “if node C is traversed, then we must traverse node H, and if node D is traversed, then we must traverse Node G” –These constraints will eliminate Paths 1, 4, 6 and 8 –We are left to consider four feasible paths: 2. first – A – B – D – E – F – G – O – last 3. first – A – B – C – E – F – H – I – N – O – last 5. first – A – B – C – E – F – H – J – K – M – N – O – last 7. first – A – B – C – E – F – H – J – L – M – N – O – last

9 Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] 1. program triangle (input, output); 2. VAR a,b,c: integer; 3. IsATriangle: boolean; 4. BEGIN 5. writeln(“Enter integers: “); 6. readln(a,b,c); 7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); 8. IF(a <b+c) AND (b < a+c) AND (c<a+b) THEN 9. IsATriangle = True; 10. ELSE IsATriangle = False; 11. IF IsATriangle) 12. THEN 13. BEGIN 14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) 15. THEN writeln(“Equilateral”); 16. IF (a=b) AND (b=c) 17. THEN writeln(“Equilateral”); 18. IF(a <> b) AND (a <> c) AND (a<>c) 19. THEN writeln(“Scalene”); 20. END 21. ELSE Writeln(“Not a Triangle”); 22. END first A B C D L E F G H A I J K last P1: A-B-D-E-G-I-J-K-Last P2: A-C-D-E-G-I-J-K-Last P3: A-B-D-L-Last P4: A-B-D-E-F-G-I-J-K-Last P5: A-B-D-E-F-G-H-I-J-K-Last P6: A-B-D-E-F-G-H-I-K-Last

10 Node B is traversed  D, E are traversed Node C is traversed  D, L are traversed Node E is traversed  F, H, J are traversed Node F is traversed  H, J are traversed Node H is traversed  F, J are traversed Node J is traversed  F, I are traversed Let’s consider the Triangle program, its CFG, and its DD-Path graph. [Jorgensen: Software Testing A Craftsman’s Approach] 1. program triangle (input, output); 2. VAR a,b,c: integer; 3. IsATriangle: boolean; 4. BEGIN A5. writeln(“Enter integers: “); A6. readln(a,b,c); A7. Writeln(“Side A is: “, a, “Side B is: “, b, “Side C is: “, c); A8. IF(a < b+c) AND (b < a+c) AND (c < a+b) THEN B9. IsATriangle = True; C10. ELSE IsATriangle = False; D11. IF IsATriangle) D12. THEN E13. BEGIN E14. IF (a=b) XOR (a=c) XOR (b=c) AND NOT((a=b) AND (a=c) F15. THEN writeln(“Equilateral”); G16. IF (a=b) AND (b=c) H17. THEN writeln(“Equilateral”); I18. IF(a <> b) AND (a <> c) AND (a<>c) J19. THEN writeln(“Scalene”); K20. END L21. ELSE Writeln(“Not a Triangle”); 22. END Logically feasible paths: P1: A-C-D-L-Last P2: A-B-D-E-F-G-I-K-Last P3: A-B-D-E-G-H-I-K-Last P4: A-B-D-E-G-I-J-K-Last first A B C D L E F G H A I J K last

11 Structured Programming Constructs s1 s2 s3 s1 s2 s4 s3 s1 s2 s5 s4 s3 s1 s3 s2 s1 s2 s1 s2 Sequence if-then-else case if-then loop-pre-test loop-post-test

12 Violations of Structured Programming Constructs s1 s2 s3 s1 s3 s2 s0 s1 s2 s3 s0 s1 s3 s2 s1 Branching into loop Branching out of loop Branching into decision Branching out of decision

13 Essential Complexity Essential complexity. This measures how much unstructured logic exists in a module (e.g., a loop with an exiting GOTO statement) The essential complexity can be computed on a program graph that has been condensed by collapsing structured programming constructs to a single node, until no more structured constructs can be found

14 Condensing A Program Graph first A B C D L E F G H A I J K last a b c a b cH L first d a dL last first e e V(G) = 1

15 Test Coverage Metrics - Revisited MetricDescription of Coverage C0C0 Every Statement C1C1 Every DD-Path C1PC1P Every predicate to each outcome C2C2 C 1 Coverage + loop coverage CdCd C 1 Coverage + every dependent pair of DD-Paths C MCC Multiple condition coverage CikCik Every program path that contains up to k repetitions of a loop (usually k=2) C stat “Statistically significant” fraction of paths C∞C∞ All possible execution paths

16 Some Fundamental Path Selection Criteria Exercise every statement or instruction at least once (statement coverage) C 0 Exercise every branch and case statement, in each direction, at least once (branch coverage) C 1 Exercise each condition in a decision (condition coverage) Exercise each condition in a decision with all possible outcomes C 1 P Exercise every compound predicate outcome (MCC) C MCC Exercise every path from entry to exit C ∞

17 Example F T F T F T T F F T Statement Coverage C 0 : SCPath1: 1-2-3(F)-10(F) SCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T) Branch or Decision Coverage C 1 : BCPath1: 1-2-3(F)-10(F) BCPath2: 1-2-3(T)-4(T)-5-6(T)- 7(T)-8-9-3(F)-10(T) BCPath3: 1-2-3(T)-4(F)-10(F) BCPath4: 1-2-3(T)-4(T)-5-6(F)-9-3(F)- 10(F) BCPath5: 1-2-3(T)-4(T)-5-6(T)-7(F)-9- 3(F)-10(F)-11-13

18 Condition Coverage Requires that each condition in a compound decision takes on all possible outcomes at least once Condition Coverage does not require that each decision takes on all possible outcomes Example: if Not (A or B) then C Setting A = True, B = False, then A = False and B = True satisfies Condition Coverage, but statement C will never be executed

19 Multiple Condition Coverage Requires that each possible combination of inputs be tested for each decision. Example: “if (A and B)” requires 4 test cases: A = True, B = True A = True, B = False A = False, B = True A = False, B = False For n conditions, 2 n test cases are needed, and this grows exponentially with n

20 Modified MCC Requires that each condition be shown to independently affect the outcome of a decision Generally, for n conditions, MC/DC requires only n + 1 test cases Example: if (A or B) then C 3 test cases: A = True, B = False (A dominates) A = False, B = True (B dominates) A = False, B = False (false outcome)

21 Examples To test if (A or B) A: TFF B:FTF To test if (A and B) A:FTT B:TFT To test if (A xor B) A:TTF B:TFT

22 Guidelines and Observations Functional testing focuses on properties that are “too far” or disassociated from the source code being tested Path testing focuses too much on the graph and not the logic of the code Path testing very useful to measure the quality of the testing (coverage criteria) Basis path testing is giving us a lower bound of how much testing is necessary Path testing is providing a set of metrics that act as cross checks to functional testing –When a program path is traversed by several functional test cases we suspect redundancy –When we fail to attain DD-Path graph coverage, we know that there are gaps is the functional test cases Coverage metrics can be used as –Setting minimum testing standards and as –Mechanism to selectively test portions of the code more rigorously than others

23 Guidelines and Observations We can distinguish between –Specified program behavior (S), – Programmed behavior (feasible paths) (P), and – Topologically feasible paths (T) S P T

24 Guidelines and Observations Region 1 contains specified behavior that is implemented by feasible paths – most desirable Every feasible path is topologically possible so the parts 2,6 of set P should be empty Region 3 contains feasible paths that corresponds to unspecified behavior. Such extra functionality must be inspected Regions 4 and 7 contain the infeasible paths. Region 4 is problematic as it corresponds to specified behavior that has almost been implemented (topologically possible, yet infeasible program paths) – coding errors Region 7 refers to unspecified, infeasible, yet topologically possible paths – potential maintenance problem – could become Region3 S P T

25 Software Testing Standards IEEE 1008 Software Unit Testing IEEE 1044 Classification for Software Anomalies ISO/IEC Information Technology--Software Life Cycle Processes ISO/IEC TR Guide for ISO/IEC (Software Life Cycle Processes) AECL CE-1001-STD REV.1 Standard for Software Engineering of Safety Critical Software BSI BS-7738 Specification for Information Systems Products Using SSADM (Structured Systems Analysis and Design Method) BSI BS Software Testing - Vocabulary BSI BS Standard for Software Component Testing DEF (Part 1)/1 The Procurement of Safety Critical Software in Defence Equipment-Requirements DIN VDE 0801 Principles for Computers in Safety-Related Systems ESA PSS Guide to the Software Engineering Standards, Issue 1 German Process-Model (V-Model) Software Life-Cycle Process Model IAEA TRS-372 Development and Implementation of Computerized Operator Support Systems in Nuclear Installations IEC Medical Electrical Equipment--Part 1: General Requirements for Safety-4.Collateral Standard: Programmable Electrical Medical Systems IEC Software for Computers in the Safety Systems of Nuclear Power Stations IEE 5 Software Inspection Handbook NCC STARTS Purchasers' Handbook--Procuring Software-Based Systems NRC NUREG/BR-0167 Software Quality Assurance Program and Guidelines RTCA DO-178B/ED-12B Software Considerations in Airborne Systems and Equipment Certification