1 Software Testing and Quality Assurance Lecture 14 - Planning for Testing (Chapter 3, A Practical Guide to Testing Object- Oriented Software)

2 Lecture Outline To learn how to analyze the risks associated with verifying the required functionality. Adequacy of test cases.

3 Risk analysis ―A tool for testing Risk analysis is part of planning and development effort. A risk ―anything that threatens the successful achievement of a project’s goals A risk is an event that has some probability of happening and, if it occurs, there will be some loss (down time, financial).

4 Risk analysis ―A tool for testing Fundamental principle for risk-based testing Test most heavily those portions of the system that pose the highest risks to the project to ensure that the most harmful faults are identified.

5 Risk analysis ―A tool for testing: risk types Project risks: include managerial and environmental risks (e.g. insufficient supply of qualified personnel). Business risks: are associated with domain-related concepts. This type of risk is related to the functionality of the program (e.g. changes on the health insurance policy).

6 Risk analysis ―A tool for testing: risk types Technical risks: include some implementation concepts (e.g. the quality of the code).

7 Risk analysis ―A tool for testing: risk analysis Risk analysis ―is a procedure for identifying risks and for identifying ways to prevent problems from becoming real. The output of risk analysis is a list of identified risks in the order of the level of risk that can be used to allocate limited resources and to prioritize decisions.

8 Risk analysis ―A tool for testing: risk analysis Risks in OO software projects are unique to the architectural features: Complex interactions among objects Complex behavior associated with a class specification Changing or evolving project requirement Complexity of a class measured by the size of its specification The number of relationships a class has with other classes

9 Risk analysis ―A tool for testing: risk analysis (cont...) Source of risks: For system testing, the various uses of the system are prioritized based on the importance to the user and proper operation of the system. Risks are also associated with the programming language and development tools that are being used to implement the software (strong typed vs. weak typed languages).

10 Risk analysis ―A tool for testing: risk analysis (cont...) Conducting the analysis: Risk analysis technique includes three tasks: Identify the risks that each use case poses to the development effort (classify as low, medium and high –may be also very high) Quantify the risk Produce a ranked list of use cases

11 Risk analysis example Brickles example You may combine the frequency and the criticality to determine which should be tested more heavily. UseRisk Level Frequen cy Criticali ty Scenario WinsMediumLowHighPlayer wins game Lose s MediumHighLowPlayer loses game

12 Risk analysis example Risk level: Conservative strategy: select the higher value. Averaging strategy: select the value between the two values.

13 A testing process: planning issues Issues that must be addressed to give a basic shape to the test process: Planning issues: Testing the models. Class testing, interaction testing, system testing, regression testing.

14 A testing process: dimensions of software testing Who performs the testing? Developers, independent testers or combination of the two. Developers may exchange code and test each other’s code. Which pieces will be tested? Test nothing, test a sample, test everything, or just the ones associated with high risks.

15 A testing process: dimensions of software testing When will testing be performed? Test every day, test components as they are developed, test all components together at the end, at special milestones. How will testing be performed? Knowledge of specification only, knowledge of specification and implementation.

16 A testing process: dimensions of software testing (cont...) How much testing is adequate? No testing, exhaustive testing (consider the expected lifetime of the software), is the software life-critical, Coverage: is a measure of how completely a test suite exercises the capabilities of a piece of software (other measures: if every LOC executed at least once, the number of requirement that is checked by the test suite).

17 A testing process: adequacy of test cases Adequacy of test cases: test the software enough to be reasonably sure that the software works as it is supposed to.

18 A testing process: adequacy of test cases Adequacy can be measured based on coverage: Coverage based on the requirements: based on what the software is supposed to do ―how many of the requirements called out in the specification are tested Coverage based on the code: based on how the software actually works ―how much of the software was executed as a result of running test suites

19 A testing process: adequacy of test cases (cont...) Functional testing (specification-based or black box testing): test cases are constructed based solely on the software’s specification and not on how the software is implemented. Structural testing (implementation- based or white box testing): test cases are constructed based on the code that implements the software.

20 A testing process: adequacy of test cases (cont...) Use some combination of both approaches.

21 A testing process: adequacy of test cases ― risk analysis (cont...) Risk analysis in the testing process is applied to determine the level of details and amount of time to dedicate to testing a component. A reasonable scale of increasing risk for components is: Prototype components Production components Library components Framework components Risk increases

22 Key points A risk ―anything that threatens the successful achievement of a project’s goals Risk analysis technique includes three tasks: identify, quantify the risk and produce a ranked list of use cases. Adequacy of test cases: test the software enough to be reasonably sure that the software works as it is supposed to.