Testing and Inspecting to Ensure High Quality Part 1: Basic Definitions Effective and Efficient Testing Lots of Personal Notes

2 Overview Testing and Inspections – two very important concepts / activities in software development. We often have ‘roles’ of testers; individual developers; for different ‘kinds’ of testing; independent test groups; quality assurance groups, etc. Quality Control; Editors; Various sizes and shapes…. Very unique to the environment within which the software is developed. We have many different types of tests too. Exist at many different levels for many different stakeholders. Some are automated; many are hands-on… Some tests test specific parts of system; others whole; Some at different stages of development; …. Many Many names!!!! Heuristic: a successful test is one that identifies a flaw. We will present a lot of very important material in these lectures.

3 Tests – just a few types of tests…. White box testing; Black Box Testing Verification and validation (V&V) Coverage Testing: Branch testing, Path testing, Statement testing Alpha testing, Beta testing Acceptance Testing; Validation Testing Unit test, Subsystem test, System test, Integrated tests Environmental System Testing; Operational Field Testing Running in parallel; Running live; Implement in pieces.

4 Organizational Considerations Deficiency Report (DIREP) Burroughs Hardware / UNISYS – years ago…. Field Assistance Branch; Customer Service Tracked, prioritized, resources allocated... Reported to senior levels of management Incident Report – Honeywell: Problem Ticket – Fix Tickets (LPS) Office of Primary Responsibility (OPR); Office of Collateral Responsibility… (OCR) Product Manager; Account Rep…

5 Basic Definitions A failure is an unacceptable behavior exhibited by a system —The frequency of failures measures the reliability —Important design objective: achieve a -very low failure rate and hence -a high reliability.

Basic Definitions: Failure Causes: A failure can result from a violation of an explicit or implicit requirement Specific —functional requirement; —non-functional requirements… 6

Basic definitions: MTBF MTTR Metrics We clearly want to increase the Mean Time Between Failures (MTBF) and reduce the Mean Time to Repair (MTTR): and damage caused by failures. MTBF and MTTR (old engineering metrics): are still considered measures of reliability. 7

© Lethbridge/Laganière Basic Definitions - continued A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures —Might take several defects to cause a particular failure —Defects can occur anywhere – requirements, design, implementation, testing, etc. —They can also occur at any time! True ‘War’ Story: “Mr. Smith said this could never happen!” An error is a slip-up or inappropriate decision by a software developer that leads to the introduction of a defect

9 Effective and Efficient Testing To test effectively, you must use a strategy that uncovers as many defects as possible. Testing really attempts to find defects! Pure & simple!

Effective and Efficient Testing To test efficiently, find the largest possible number of defects, using the fewest possible tests. Testing costs time, money and other resources! Testing is not cheap! It is quite expensive! This is where the problems come in. What is the expected return on investment? There are many kinds of tests designed to uncover different kinds of defects. 10

11 Effective and Efficient Testing – more… Testing must be efficient. So what do we test?? We cannot test everything. (Exhaustive Testing is not possible – proven mathematically) Rather, we design different kinds of tests for ‘sufficient coverage.’ All kinds of tests: Many Categories Some are lumped into Coverage Tests. statement coverage branch coverage path coverage….. Others tests focus on creating appropriate test values (Equivalence Testing)

Verification and Validation (V&V) Developers usually do their own testing first (verification) —Usually unit testing; testing within development team. —May involve subsystem testing, integrated testing. … —But, in general, the developers are testing their products. —Generally their testing is white box testing, but can be black box testing. End users test system against requirements (validation) —This is essentially called black box testing, although end users may choose to call it validation testing.

12 Black-Box Testing Testers provide the system with inputs; observe outputs They can see none of: — Source code — Internal data (no algorithms or data structures, …) — Design documentation describing the system’s internals They simply do inputting based on requirements and observe outputs. Testing Charge: Often depends on the kind and level of testing and what the ‘charge’ of the testing group (organization) really is.

13 Black-Box Testing - more Kind of testing end-users most frequently undertake – for obvious reasons. Sometimes called validation testing / acceptance testing - IF done when system is declared ‘finished.’ Black box testing may sometimes be called other things (alpha testing; beta testing…) but these tests (alpha and beta) usually have other connotations associated with them… Black Box Testing is designed to show functionality – and often (sometimes) satisfaction of non-functional requirements. – (such as load considerations; simultaneously users, distribution, more)

15 Glass-Box Testing – White-Box Testing Also called ‘structural’ testing; Developers and Testers have access to the system design They can —Examine the Design and these Documents —View the code for Standardization / conformance! —Observe at run time the steps taken by algorithms and their internal data Individual programmers often informally employ glass-box testing to verify their own code and functionality Developers (using tools) can tell what parts of their programs are executed the most (most time spent in them) and then these may be candidates for optimization….

16 Glass-Box Testing - White Box Testing Unit Testing Developers are always responsible for their own ‘modules’. modules (units) = programs, classes, subsystems, etc. Developers ‘verify’ their products. (may undertake both black box and white box testing…) Developer programs implement the design Programs satisfy functional requirements, etc… A lot of work to undertake ‘unit testing.’ Hazardous though, since they may be required to simulate inputs, outputs, files passed, parameter values passed, etc…. Many kinds of unit testing…

17 Glass-Box - White Box Testing - More Path Testing – exhaustive; impossible; there is an infinite number of paths in a non-trivial program. Branch testing – Edge testing – the most feasible Design a test so that all of the edges of a node are executed. Often shown via flowgraphs. (ahead) What do we mean by ‘edge?’ (all ‘else’ branches taken; all ‘then’ branches executed…) Consider a flowgraph of an algorithm (two slides up) …. Emphasis will be on control and coverage.

18 White Box: Branch Testing - Flow Graphs Want minimum number of tests assuring high degree of reliability. Know we cannot test all paths in a non-trivial program. Want a ‘representative set.’ Given a flow graph (next slides), there are a couple of formulas that will provide the minimum number of tests. They are: (number of nodes – number of edges + 1) Number of Regions + 1 ….(cyclomatic complexity) Number of Regions is considered to be a measure of complexity.

19 Consider: The number of possible paths through a subprogram is equal to the number of regions in the subprogram's flowgraph. The flowgraph below is for a SquareRoot function and has regions numbered, as shown. Can see there are four regions, as numbered, plus the outer region, or 5 in total.

20 This indicates that there are five possible paths through the flowgraph which can be described by listing the sequence in which the nodes must are visited. The five paths in the SquareRoot flowgraph are as follows. etc…

21 White Box: Equivalence Class Testing This is a Bigee! It is inappropriate to test by brute force, using every possible input value —Discuss; Takes a huge amount of time —Is impractical and is pointless! Far better: divide the possible inputs into groups which you believe will be treated similarly by all algorithms. —Such groups are called equivalence classes. —Remember Computational Structures?? (COT 3100??) —What does the equivalence relation, ‘equivalence mod 4’ do for you? (Be nice!!) What does imposing this relation on the set of positive integers do?

22 Equivalence Class Testing Recall ramdomizing in hashing for tables and file structures If we used an ‘Equivalence mod 4’, we essentially divided the set of all non- negative integers by four and only take their ‘remainders’ 0, 1, 2, and 3. If a numeric key was used as a primary or relative key, this approach was often used to arrive at a disk address or an address in a hash table. Thus ‘Equivalence mod 4’ effectively partitions the set of all integers into one of five disjoint equivalent classes. ALL division operations result in one of these remainders – and those having the same remainder (members of the same equivalence class) need only require one of its members to be tested to represent ALL values that fall into this equivalence class.. We applied this equivalence relation to the set of integers and obtained a set of equivalence classes the Union of which constitutes the original set of Integers. —EC1 U EC2 U … ECn = set of I+ (non-negative)

23 Example of Equivalence Classes —A tester need only to test one member of an equivalence class rather than ALL numbers of that class. —The tester has to -understand the required input, -appreciate how the software may have been designed Example: —Valid integer input is a month number (1-12) —Equivalence classes are: [-∞..0], [1..12], [13.. ∞] These are the three equivalence classes. In Equivalence Class Testing, we select an input from each of the equivalence classes as inputs to testing. This may not be sufficient testing, but it is a good start….

24 More on Equivalence Class Testing To test for valid integers ranging from 1 through 12, we have three equivalence classes as indicated: —Integers less than 1; —Integers between 1 and 12 inclusive —Integers greater than 12. Is this enough? Equivalence Class Testing cannot handle all kinds of testing: What if there is more than a single value we are testing?

25 Combinations of Equivalence Classes Combinatorial explosion means that you may not be able to realistically test every possible system-wide equivalence class. Let’s say there are 4 inputs each of which have 5 possible values. This means that are 5 4 (i.e.625) possible system- wide equivalence classes. While you should first make sure that at least one test is run with every equivalence class of every individual input, you should also test all combinations where one input is likely to affect the interpretation of another. Let’s look at another type of test….

26 Compatibility Tests; Associativity Tests Compatibility tests; Associativity tests. —Example of Associativity Test: —Example from a Manpower Project in the USAF. -Field/attribute: Rated Performance Indicator (rpi); text field; permissible values: 0 or other integer such as 1 -Rule: If the rpi input value for an individual had a value of non- zero ( individual had to be on an aircrew), then the ‘rank’ attribute for this individual (s/he must be an officer), must be in the range between O1-O10 (and could NOT be in the range E1-E9). -(O  officer rank; E  Enlisted rank) —Example of Compatibility Test: -Range tests: If rank = ‘O4’, » salary must fall between $nnnn and $nnnnnn. Many others and many different types of tests.

27 Testing at Boundaries of Equivalence Classes Boundary Value Testing More errors in software occur at the boundaries of equivalence classes than at any other place!! AMEN!!! The idea of equivalence class testing should be expanded to specifically test values at the extremes of each equivalence class —E.g. The number 0 often causes problems E.g.: If the valid input is a month number (1-12) —Test equivalence classes as before -Test -24; 6, +24, (representative values) —Test boundaries: 0, 1, 12 and 13 as well as very large positive and negative values —Are only integers allowed? (perhaps another test)