1. Object-Oriented Software Engineering Practical Software Development using UML and Java Chapter 10: Testing and Inspecting to Ensure High Quality Part 1: Basic Definitions Effective and Efficient Testing Lots of Personal Notes

2. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality2 Testing and Inspections – two very important concepts / activities in software development. We often have roles of testers; individual developers; members of development teams; independent test groups; quality assurance groups, etc. Quality Control; Editors; … We have many different kinds / types of tests too. Exist at many different levels for many different stakeholders. Some are automated; many are hands-on… Some test parts of system; others whole; Some at different stages of development; …. Many Many names!!!! Note: a successful test is one that identifies a flaw. We will present a lot of very important material in these last lectures.

3. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality3 Tests White box testing; black box testing Verification and validation (V&V) Branch testing, path testing, statement testing Alpha testing, beta testing Acceptance testing Unit test, subsystem test, system test, integrated system test, … Independent test groups Environmental system testing Operational Field Testing Running in parallel; running live;

4. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality4 Organizational Considerations DIREP Deficiency Report Tracked, prioritized, resources allocated... Incident Report Problem Ticket – Fix Ticket Office of Primary Responsibility (OPR); Office of Collateral Responsibility… (OCR) Product Manager…..

5. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality5 10.1 Basic definitions A failure is an unacceptable behaviour exhibited by a system —The frequency of failures measures the reliability —An important design objective is to achieve a very low failure rate and hence high reliability. —A failure can result from a violation of an explicit or implicit requirement —An extreme and easily understood kind of failure is an outright crash!! We clearly want to reduce the frequency of failures (MTBF) and damage caused by failures. (Mean Time Between Failures) We measure failures and time to repair (MTTR) as measures of reliability. (Mean Time to Repair)

6. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality6 A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures —It might take several defects to cause a particular failure —Note also that a defect can occur anywhere – requirements, design, implementation, testing, etc. True Story: “Mr. Smith said this couldn’t happen!” An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect

7. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality7 10.2 Effective and Efficient Testing To test effectively, you must use a strategy that uncovers as many defects as possible. Testing really attempts to find defects! Pure and simple! To test efficiently, you must find the largest possible number of defects using the fewest possible tests. This is where the problems come in. There are many kinds of tests designed to uncover different kinds of defects. Testing must be efficient. We cannot test everything. Rather, we design different kinds of tests for sufficient ‘coverage.’ All kinds of tests: Some are lumped into categories called Coverage Tests. statement coverage branch coverage path coverage….. We will discuss ahead.

8. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality8 Black-box testing Testers provide the system with inputs and observe the outputs They can see none of: —The source code —The internal data —Any of the design documentation describing the system’s internals Simply putting in inputs and observing outputs. Bouncing these against the requirements. This is the kind of testing that end-users most frequently undertake – for obvious reasons. Called validation testing / acceptance testing - IF done when system is declared ‘finished.’ —May be called other things (alpha testing; beta testing…) Black box testing is designed to show functionality and (often to a lesser degree at this time – but not always…) satisfaction of non- functional requirements. (performance, load…) Generally, black box testing is designed to verify functionality!!

9. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality9 Who does this? Clearly, end users. BUT: developers must do their own ‘verification’ of functionality before turning this loose! Developers: ‘verification’ Customers: ‘validation’ Hence V&V.

10. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality10 Glass-box testing – White-box testing Also called ‘white-box’ or ‘structural’ testing Developers and Testers have access to the system design They can —Examine the design documents —View the code —Observe at run time the steps taken by algorithms and their internal data Individual programmers often informally employ glass- box testing to verify their own code.

11. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality11 More – White Box testing  Unit Testing Developers are always responsible for their own modules. Developers ‘verify’ their products. Usually called ‘unit test’ But there are many kinds of unit testing…

12. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality12 Glass-Box (White Box) Testing - More Path Testing – exhaustive; impossible; there is an infinite number of paths in a non-trivial program. Branch testing – Edge testing – the most feasible Design a test so that all of the edges of a node are executed. What do we mean by ‘edge?’ Consider a flowgraph of an algorithm…. (Think of a program flowchart)

13. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality13 White box: Flow Graphs Want a minimum number of tests that assures a high degree of reliability. Know we cannot test all paths in a non-trivial program. Want a ‘representative set.’ Given a flow graph, there are a couple of formulas that will provide the minimum number of tests (like number of nodes – number of edges + 1) (?) and others. One famous way is the Cyclomatic Complexity of a Flowgraph. Defined to be: number of regions of the graph + 1 The ‘number’ of regions is called the cyclomatic complexity of the routine.

14. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality14 Equivalence Class Testing This is a Bigee! It is inappropriate to test by brute force, using every possible input value —Discussed; Takes a huge amount of time —Is impractical and is pointless! Far better: divide the possible inputs into groups which you believe will be treated similarly by all algorithms. —Such groups are called equivalence classes. Recall ramdomizing in File Structures (hashing…). If we used an Equivalence mod 4, we essentially divided all ‘remainders’ into one of five sets; those with remainders 0, 1, 2, 3, and 4. ALL divisions resulted in one of these remainders. We applied this equivalence relation to the set of integers and obtained a set of equivalence classes the Union of which constitutes the original set of Integers. —EC1 U EC2 U … ECn = set of I+

15. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality15 Examples of equivalence classes —A tester needs only to run one test per equivalence class —The tester has to -understand the required input, -appreciate how the software may have been designed Example: —Valid input is a month number (1-12) —Equivalence classes are: [-∞..0], [1..12], [13.. ∞] These are the three equivalence classes. In Equivalence Class Testing, we select inputs from each of the equivalence classes and include a representative value from EACH equivalence class.

16. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality16 More on Equivalence Class Testing So, for valid integers ranging from 1 through 12, equivalence classes are as indicated (above). Integers less than 1; Integers between 1 and 12 inclusive Integers greater than 12. (Is this enough?) What if there is more than a single value we are testing?

17. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality17 Combinations of equivalence classes Combinatorial explosion means that you cannot realistically test every possible system-wide equivalence class. —If there are 4 inputs with 5 possible values there are 5 4 (i.e.625) possible system-wide equivalence classes. While you should first make sure that at least one test is run with every equivalence class of every individual input, you should also test all combinations where an input is likely to affect the interpretation of another. Compatibility tests; Associativity tests. —Examples: field: non-zero rpi,  rank had to be officer (grade O1 through O10) —Range tests: If grade = O4, salary must fall between $nnnn and $nnnnnn. —Testing multiple fields – all white box!!!

18. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality18 Testing at boundaries of equivalence classes – Boundary Value Testing More errors in software occur at the boundaries of equivalence classes than at any other place!! AMEN!!! The idea of equivalence class testing should be expanded to specifically test values at the extremes of each equivalence class —E.g. The number 0 often causes problems  E.g.: If the valid input is a month number (1-12) —Test equivalence classes as before -Test -24; +24, and a 6 (representative values) —Test boundaries: 0, 1, 12 and 13 as well as very large positive and negative values

19. © Lethbridge/Laganière 2001 Chapter 10: Testing and Inspecting for High Quality19 Coming Defects in ordinary algorithms…. There are many of these. Read carefully.