Risk-Based Management and Testing

2 This is risk-based testing(J. Bach) Make prioritized list of risks Perform testing that explores each risk As risks evaporate and new ones emerge, adjust your test effort to stay focused on the current crop

3 What questions testing must answer? Does the product meet quality expectations? Is the application ready to be released? What are we risking if we release now? What if 20,000 users simultaneously start to download from our website?

4 Meeting the demand with limited resources The most important problems are found Requirements with the biggest impact to users are tested first Problems are detected early Problems with most potential re-work are found first It is much more important “what” you test than “how much”

5 What is the goal of testing? To address the risks To show the benefits (requirements being met) NOTE: risks by definition is something that threatens the benefits of the project

6 What is risk? Risk is a problem that might happen ISTQB: A factor that could result in future negative consequences; usually expressed as impact A risk is the chance of an error occurring (chance of failure) related to the damage expected when this error does occur

7 Why we should asses the risk? The risk helps prioritization – high risk means high priority Having good priorities set ensures the best possible result if testing is stopped before the planned end date

8 Why risk-based strategy will help?

9 3 Dimensions of the risk in software testing A way the program could fail How likely is that program would fail in that way What are the consequences of that failure

10 Risks and Contingencies Plan(IEEE Std )

11 Risk described as a formula Risk = Chance of failure x Damage  Chance of failure = likelihood  Damage = Impact

12 Determine the way the program could fail Ask following questions:  What the failure will look like?  How do I detect this failure?  How expensive is to search for the failure?  Who would be impacted by that failure?  How serious would that failure be?  How expensive would it be to fix?

13 Determine the damage Major factors  Critical Areas determined by cost and consequences of the failure  Catastrophic/Damaging/Hindering/Annoying  Visible areas – where many users will experience the errors  Usage frequency - how much you use a feature or function  Unavoidable/Frequent/Occasional/Rare

14 Determine the Chance of Failure Complexity of the functionality Changed Areas New functionality New technology, tools, methods used Number of people in team Time Pressure Geographical Spread Insufficient involvement of users ……….

15 A software product is set of Functionalities/Components Quality Characteristics  Usability  Performance  …..

16 Product Risk Assessment – High Level Potential Risk (High – 5, Medium – 3, Low – 1, N/A – 0) Feature 1Feature 2 Used heavily by users55 With complex logic/functions35 Feature is being ‘fixed’ often13 High availability required00 Consistent performance35 New development tools/languages used03 Many interfaces00 Inexperienced developers (or developers changed)51 Not enough user involvement55 Large development team11 Completely New15 Developed under extreme pressure11 Most important to stakeholders35 Many defects reported in previous versions30 Risk Score2839

17 Quiz How is risk related with time estimations for each feature?

18 Likelihood – Impact Analysis ATM System LikelihoodImpactPriorityMitigation FeaturesAttributes Withdraw cashHi 6Code inspect Deposit cashMedHi5 Check balanceLoMed3 Transfer fundsMed 4 Purchase stampsHighLow4 Make a loan payment LowMed3 UsabilityMedHi5Early user review PerformanceLowMed3 SecurityMedHi5

19 Quiz Who is the best source of information about the likelihood of a bug? Who is the best source of information about the impact of a bug?

20 Answer Likelihood – best source of info are developers, because the likelihood is determined by complexity, number of interfaces etc. Impact – best source of info are users or customers because impact is driven by business issues

21 Master Test Planning Req Insp Design Review Compo nent lvl Integrati on lvl System lvl Relative importance Security+++5 Continuity- Reusability- Efficiency- Functionality Performance Portability+++10 Flexibility- Userfriendlin ess %

22 Test Level Test Planning Sub System 1 Sub System 2 Sub System 3 Sub System 4 Relative importance Securityxxx5 Continuity- Reusability- Efficiency- Functionalityxxxxxxxx50 Performancexxx15 Portability10 Flexibility- Userfriendlin ess xxxxxxx %

23 How will you test a product you do not know? First test the whole system but shallow  Cover main business scenarios  Cover few important failure situations After the test determine the features with most defects found  Give the highest priority to them for the next round During the next round go deep into highest priority areas

24 Planning Risks and Contingencies Planning risks are unscheduled events or late activities that may jeopardize the testing schedule:  Budget  Staff Availability  Training needs  Delivery dates

25 Possible Contingencies: Reduce the scope Delay implementation Add resources Adjust quality level

26 Readings Systematic Software Testing, Rick D. Craig and Stefan P. Jaskiel Software Testing Fundamentals: Methods and Metrics, by Marnie L. Hutcheson A Risk-Based Test Strategy, Dr. Ingrid B. Ottevanger A Risk-Based Test Strategy A Strategy for Risk-Based Testing, Stephane Bessone A Strategy for Risk-Based Testing Risk Based Testing, Hans Schaefer Risk Based Testing Risk-Based Testing, Cem Kaner and James Bach Risk-Based Testing Heuristic Risk-Based Testing, James Bach Heuristic Risk-Based Testing