Risk-Based Testing – An Overview Assurance with IntelligenceSlide 1 Paul Gerrard Gerrard Consulting 1 Old Forge Close Maidenhead Berkshire SL6 2RD UK e: w: t:

I Why Risk-Based Testing? II Introduction to Risk-Management III Risk and Test Objectives IV Designing the Test Process V Project Intelligence, Test Strategy and Reporting V1 Close, Q&A Here’s the commercial bit: - This material is based on: - Risk-Based E-Business Testing, Gerrard and Thompson, Artech House, Visit for more information. Agenda Slide 2Assurance with Intelligence

Why Risk Based Testing?

Requirements Functional Specification Physical Design Program Specification User Acceptance Test System Test Integration Test Unit Test V-Model Is there ever a one-to-one relationship between baseline documents and testing? Where is the static testing (reviews, inspections, static analysis etc.)? Slide 4Assurance with Intelligence

Slide 5 “Traditional” approach Test stage Consider Schedule, Environments, Timescales etc. Acceptance System Test DevTest Methodology Build and Execute tests Not again! Not focused Not Done Stakeholder Involvement Are these faults really severe? Too detailed To understand We have to Trust them Slide 5

Sequence of decisions - Stages  responsibility  capability  objectives Guidance to developers and testers - None, except generic, text book mantras - “demonstrate software meets requirements” Input of stakeholders - Only when system/acceptance tests reveal problems - Far too late! Decision making - Timescale driven in early stages - Crisis driven towards the end - Unsatisfactory all round. Problems with tradition Slide 6Assurance with Intelligence

Slide 7 Write Requirements Specify System Design System Test the Requirements Test the Specification Test the Design Unit Test Acceptance Test System Test Integration Test Install System Build System Build Software Write Code W-Model Slide 7Assurance with Intelligence

Slide 8 risk-based test reporting assess product risks Decide Risk-based testing Plan assess product risks define test objectives test techniques, products to test Stakeholder Involvement responsibility estimation process Schedule focused test design and execution Implement

If every test aims to address a risk, tests can be prioritised by risk It’s always going to take too long so… - Some tests are going to be dropped - Some risks are going to be taken Proposal: - The tester is responsible for making the project aware of the risks being taken - Only if these risks are VISIBLE, will management ever reconsider. Risk-based test planning Slide 9Assurance with Intelligence

Enough testing has been planned when the stakeholders (user/customer, project manager, support, developers) approve: TESTS IN SCOPE - They address risks of concern and/or give confidence THE TESTS THAT ARE OUT OF SCOPE - Risk is low OR these tests would not give confidence The amount and rigour of testing is determined by CONSENSUS. How much testing is enough? Slide 10Assurance with Intelligence

Even penguins know how to manage risk! Slide 11Assurance with Intelligence

Do nothing! Pre-emptive risk reduction measures - information buying - process model - risk influencing - contractual transfer Reactive risk reduction measures - contingency plans - insurance The risk that’s left is the residual risk. Risk response planning Where testing fits in Slide 12Assurance with Intelligence

Risk and Test Objectives

If we focus on risks, we know that bugs relating to the selected mode of failure are bound to be important. If we focus on particular bug types, we will probably be more effective at finding those bugs If testers provide evidence that certain failure modes do not occur in a range of test scenarios, we will become more confident that the system will work in production. Why use risks to define test objectives? Slide 14Assurance with Intelligence

Risks and test objectives - examples

Other test objectives relate to broader issues - contractual obligations - acceptability of a system to its users - demonstrating that all or specified functional or non-functional requirements are met - non-negotiable test objectives might relate to mandatory rules imposed by an industry regulatory authority and so on Risk assessment might miss something, or de-scope something important Generic test objectives - ‘catch all’ measure – e.g. all requirements coverage - complete the definition of your test stages. Risk-based test objectives are usually not enough Slide 16Assurance with Intelligence

Generic test objectives Slide 17Assurance with Intelligence

Designing the Test Process

Slide 19 Risk Identification Consult business, technical staff Prepare a draft register of risks Risk Analysis Risk Response Test Scoping Test Process Definition Discuss risks Assign probability and consequence scores Calculate exposure Formulate test objectives, select test technique Document dependencies, requirements, costs, timescales for testing Assign Test Effectiveness score Nominate responsibilities Agree scope of risks to be addressed by testing Agree responsibilities and budgets Draft the test process from the Test Process Worksheet Complete test stage definitions Tester Activity Workshop Tester Activity Review and Decision Tester Activity Master Test Planning process Assurance with IntelligenceSlide 19

Test process worksheet Slide 20Assurance with Intelligence

Slide 21 Test products through the lifecycle Assurance with IntelligenceSlide 21 initial risk assessment test objectives test stages test process definition master test planning test plan/ procedures test specification test log test execution release risk assessment test results analysis today Plann ed end Progress through the test plan Residual Risks star t

Project Intelligence, Test Strategy and Reporting Slide 22Assurance with Intelligence

Slide 23 PI and the project lifecycle Assurance with IntelligenceSlide 23 Project Initiation Project Planning PI Management Acceptance Goal Assessment Key: Stakeholder Involvement/Project Assurance/Governance Project planning and initiationProject Intelligence Activities Development activitiesReview and Test activities Test Strategy Project Intelligence Planning Results Chain Analysis PI Strategy

Slide 24 PI Strategy overview Assurance with IntelligenceSlide 24 Risks Coverage goals Business goals PI DriversAss. Obj. Project Phase Reqs DesignBuildIntegSystestUATTrialProd. Objectives for each test phase are easily identified

Slide 25 PI Process Overview – designed to handle change Assurance with IntelligenceSlide 25 Test Process Management Project Risk Management PI Reporting Test Phase Coverage Goals Change Goals and Risks 3 Project or Process Risks 4 Evaluation & Analysis 13 Impact Analysis 1 Requirements Analysis/ coverage 2 Classification 5 Product Risks Out of Scope 7 Goals Outstanding Risks Outstanding Coverage Outstanding 12 Goals Achieved Risks Addressed Coverage Achieved 6 Allocation to Test Phase 9 Run Test 10 Identify Regression Test 11 Run Regression Test 8 Define/ Design Tests Risk is not product-related, or is programme-critical Product risk Risk will not be tested (no test objective) Test objective defined All new goals/risks are “outstanding” initially Tests have revealed a new risk or changes to a risk Change affects requirements (and tests) Change requires regression tests Failed tests must be repeated when fixes received New risk identified

Slide 26 Risk and goal-based reporting Assurance with IntelligenceSlide 26 Risks Coverage goals Business goals PI DriversAss. Obj. Project Phase Reqs DesignBuildIntegSystestUATTrialProd. Progress towards goals is clearly seen. Outstanding risks are highly visible.

Slide 27 Risk-based reporting Assurance with IntelligenceSlide 27 Progress through the test plan today Planned end residual risks of releasing TODAY Residual Risks start all risks ‘open’ at the start

Slide 28 Goal based test reporting Assurance with IntelligenceSlide 28 Open Closed Risks Open Closed Open Objective Goal Benefits available for release ObjectiveBenefit Closed

Our testing is good if it provides: - Evidence of the benefits delivered - Evidence of the CURRENT risk of release - At an acceptable cost - In an acceptable timeframe Good testing is: - Knowing the status of benefits with confidence - Knowing the risk of release with confidence. How good is our testing? Slide 29Assurance with Intelligence

RBT approach helps stakeholders: - They get more involved and buy-in to the approach - They have better visibility of the test process RBT approach helps testers - Approval to test against risks in scope - Approval to not test against risks out of scope - Clearer test objectives upon which to design tests RBT approach helps developers - Specifies their responsibility for testing in detail - “No hiding place”. Risk-based test approach: planning Slide 30Assurance with Intelligence

RBT approach helps stakeholders: - They have better visibility of the benefits available and the risks that block benefits RBT approach helps management: - To see progress in terms of risks addressed and benefits that are available for delivery - To manage the risks that block acceptance - To better make the release decision. Risk-based test approach: execution and reporting Slide 31Assurance with Intelligence

Risk-Based Testing Any Questions? riskbasedtesting.com gerrardconsulting.com uktmf.com