Stoimen Stoimenov QA Engineer SitefinityLeads,SitefinityTeam6 Telerik QA Academy Telerik QA Academy

  Risk and Testing – Main Concepts   Product Risks   Project Risks   Risk-Based Testing   Risk Management   Risk Identification   Risk Analysis (Assessment)   Risk Control 2

Main Concepts

 Risk  The possibility of a negative or undesirable outcome or event  Any problem that may occur would decrease perceptions of product quality or project success 4

 Two main types of risk are concerned  Product (quality) risks  The primary effect of a potential problem is on the product quality  Project (planning) risks  The primary effect is on the project success  Factors relating to the way the work is carried out 5

 Not all risks are equal in importance  Factors for classifying the level of risk:  Likelihood of the problem occurring  Arises from technical considerations  E.g. programming languages used, bandwidth of connections, etc.  Impact of the problem in case it occurs  Arises from business considerations  E.g. financial loss, number of users affected, etc. 6

RISK Impact(damage) Likelihood (Probability of failure) Use frequency Lack of quality 7

 Effort is allocated proportionally to the level of risk  The more important risks are tested first  Risk management  The more important the risk, the tighter the control  Test results and project status are reported in terms of residual risks  E.g. which tests have not yet been run or have been skipped 8

 Risk management does not happen only once in a project  Risk must be periodically reevaluated  Based on new information  Strategic changes might be needed  reprioritizing tests and defects  reallocating test effort  taking other test control activities 9

 Testing can be described as a form of insurance  When do you buy an insurance?  When you are worried about some potential risk  Risk-based testing relies on qualitative analyses  Statistically valid data for quantitative analysis is usually not available 10

 Risk-based testing uses risk to prioritize and emphasize the appropriate tests during test execution 11

 What is a product risk?  The possibility that the system or software might fail to satisfy some reasonable customer, user, or stakeholder expectation  Also referred to as "quality" risk 13

 What does "unsatisfactory software" mean?  Omitted key functionality  Unreliable and frequently fail to behave normally  Might cause financial or other damage to users  Poor software characteristics  Low security, usability, maintainability or performance  Poor data integrity and quality 14

 Organizational factors:  Skill, training and staff shortages  Complexity of the project team / organization  Inadequate expectations or improper attitude toward testing  E.g., not appreciating the value of testing 16

 Technical issues:  Ambiguous, conflicting or non-prioritized requirements  Excessively large number of requirements  High system complexity  Quality problems with the design, the code or the tests  Insufficient or unrealistic test environments 17

 Supplier issues:  Failure of a third party  Contractual issues 18

Risk-Based Testing

 What is Risk-based testing?  An approach to testing that aims to:  Reduce the level of product risks  Inform stakeholders on their status  Starts in the initial stages of a project  Involves the identification of product risks and their use in guiding the test process 20

 Risk management includes three primary activities:  Risk identification  Risk analysis  Assessing the level of risk  Risk control  Mitigation  Contingency  Transference  Acceptance 22

 Product and quality risks can be identified  Expert interviews  Project retrospectives  Risk workshops and brainstorming  Checklists  Calling on past experience 24

 Include representatives of all (possible) stakeholders in risk identification  The broadest range of stakeholders will yield the most complete, accurate, and precise risk identification 25

 Risk identification techniques can look in two directions:  "Downstream"  Identify potential effects of the risk item if it becomes an actual negative outcome  "Upstream"  Identify the source of the risk 26

 Risk analysis (assessment) involves the study of the identified risks  Categorize each risk item appropriately  Important for complex projects  Assign each risk item an appropriate level of risk  Involves likelihood and impact as key factors 28

 Complexity of technology and teams  Personnel and training issues  Supplier and vendor contractual problems  Geographical distribution of the development organization  E.g., out-sourcing 29

 Legacy (established) versus new designs and technologies  The quality (or lack of quality) in the tools and technology used  Bad managerial or technical leadership  Time, resource, and management pressure  Especially when financial penalties apply 30

 Lack of earlier testing and quality assurance tasks in the lifecycle  High rates of requirements, design, and code changes in the project  High defect rates  Complex interfacing and integration issues 31

 Potential damage to image  Loss of customers and business  Potential financial, ecological, or social losses or liability 32

 Civil or criminal legal sanctions  Loss of licenses, permits, etc.  The lack of reasonable workarounds  The visibility of failure and the associated negative publicity 33

 Quantitatively  Using numerical ratings for both:  Likelihood (usually percentage)  Impact (often a monetary quantity)  Both can be calculated to a common risk index  Qualitatively  E.g., very high, high, medium, low, very low 34

 Risk control has four main options:  Mitigation  Taking preventive measures to reduce the likelihood and/or the impact of a risk  Contingency  Where we have a plan or perhaps multiple plans to reduce the impact if a risk should it occur 36

 Risk control has four main options:  Transference  Getting another party to accept the consequences of a risk should it occur  Accepting (ignoring) the risk  A final option 37

 Various techniques can be used for risk control:  Choosing an appropriate test design technique  Reviews and inspection  Reviews of test design 38

 Various techniques can be used for risk control:  Setting appropriate levels of independence  For the various levels of testing  Using the most experienced person on test tasks  Using strategies for confirmation testing (retesting) and regression testing 39

Questions?