1. SOFTEC 2011 Risk Based Testing
Anne Mette Hass
Version 1.1 release : SSM added. Change in map (Morocco)

2. Risk-Based Testing Introduction Risk Management – Risk Identification
Risk Management – Risk Analysis
Risk Management – Risk Mitigation

3. You can’t test everything
In practice a product can have astronomical numbers of input combinations
100% test is not possible
Software testing is (therefore) sample control
Testing should be directed by
risks
willingness to run risks

4. Alternatively, a risk can be defined as:
What is a risk?
A risk is defined as:
”The possibility of realizing an unwanted negative consequence of an incident.”
Alternatively, a risk can be defined as:
“A problem which has not materialized yet and possibly never will.”
Risici can opdeles in to kategorier
Generisk: Dvs. risici der gælder for alle typer projecter.
projectspecifik: Dvs. risici der gælder for dette specifikke project.
Risici can yderligere opdeles in fire typer:
Teknisk risiko: Dvs. risici forbundet med den teknologi, der indgår in the project or productet.
Ressourcerisiko: Dvs. risici forbundet with de personer, som deltager in the project.
method risiko: Dvs. risici forbundet with den måde we gør tingene on!
Ekstern risiko: Dvs. risici der kommer udefra and påvirker the project!
Ofte forveksles risici with problemer.
a problem can defineres som a risiko, der har a 100% sandsynlighed for to indtræffe - the is erkendt, to the vil ske! 4

5. Life is full of risks
If a risk is realised, this will result in disappointment – or what’s worse
In software development many may be effected e.g.:
the company management
the project management
the project participants
the customer
the users
other stakeholders

6. Risks hit in various places
Risk categories include:
the business – this is the responsibility of management
processes
the project
the product – this is the testers’ primary focus!

7. Project risks threaten the completion of the project people
accessibility, competences, knowledge, personalities
time
money
surroundings
processes, tools, hardware
external interfaces
customer / supplier relationship

8. functional requirements non-functional requirements design programming
Product risks
Product risks threaten product quality in relation to the customer’s requirements and needs
functional requirements
non-functional requirements
design
programming
Here are the defects
Testing also involve risks
Testing can mitigate risks

9. Risk types
Risks may be related to
security of people
finances
safety of data and systems
the environment
functional and non-functional factors
technology
policy (at various levels)‏

10. Risk has two dimensions
A risk is affected by:
the probability of the problem arising
the effect in the event that the problem does occur
Risk level = probability x effect
No probability or no effect = 0 risk
Complete certainty = not a risk, but a problem

11. Test management can handle project risks
Test planning entails identifying (test) project risks
The test sub-process requires
planning and
managing
so that the most important project risks
are reduced or
prevented
Resources
Time
Quality

12. Test management can handle product risks
Risk-based test planning should be based on the identification and analysis of product risks.
Test planning should
target testing
different test techniques for different types of product risks
prioritise testing
the higher the risk, the earlier testing should occur
the higher the risk, the more time should be allocated for testing
Product risks (probability) are reduced when the test reveals failures and the defects are corrected

13. Test management informs about risks
Test management entails reporting test results.
Test results can tell about
test project progress
which product risks
have been tested for (and re-tested or as yet not re-tested!)‏
still outstanding (as yet untested or given up on)‏
which processes are weak
Process risks can be reduced by analysing failure reports

14. Risk management consists of: risk identification – what might happen?
risk analysis – how bad is it and what’s worst?
risk mitigation – what should be do?
risk follow up – how do risks develop?
Risikostyring is kendt from many other fagområder, how the indgår som a almindelig arbejdsrutine. Eksempler herpå is from den finansielle verden, how man laver risikoevaluering and styring on e.g. valutakurser. or the can være drift of kraftværker, how man laver nødprocedurer for, hvordan man forholder sig ved driftsforstyrelser, etc.
purposeet with risikostyring is to identificere de handlinger, som we can foretage in dag, der øger chancen for to the project can gennemføres succesfuldt inden for de givne rammer.
Risikostyring består grundlæggende of tre ting:
· Risikoidentifikation and kvantificering.
· Risikominimerende planning.
· Risikoopfølgning.
Risikostyring foretages of projectgroupn, with a projectleder som den overord-nede responsibilitylige for opfølgning and rapportering. Den indledende planning foretages of a group on 3-5 participants with kendskab to projectets arbejdsområde. 14

15. Risk management must be continuous
Risk management should start as early as possible and should finish with the product
risk identification and analysis (RIA)
test planning based on RIA
testing on the basis of the risk-based plan
repeat RIA on the basis of test execution results
re-plan based on the newest RIA
test in accordance with the newest plan
etc……

16. Stakeholder involvement is necessary
Experts, and the usability, safety, reliability, performance and portability responsible person
Technology, product
Analysts, Designers, Programmers, Testers
Production Managers
Operational Managers
Configuration Management Responsible Managers
Quality Assurance Managers
Technology, development
Marketing personnel
Sales personnel
Employees from the relevant support function
User representatives
Future users
Users of the existing system
User representatives from the various types of user groups
Product Manager
Business Analysts
Financial Managers
The company
Stakeholder representatives
Perspective
Involve everyone in risk identification and analysis activities

17. Risk-Based Testing Risk Management – Risk Identification Introduction
Risk Management – Risk Analysis
Risk Management – Risk Mitigation

18. Risk identification
Risk identification
means finding out what can go wrong – and writing it down, for example using a risk template
should be performed as early as possible
should be re-evaluated regularly during the entire project
provide the basis for risk analysis
provide the basis for risk mitigation

19. Examples of product risks - 1
It may be possible that …
the customer bases a business decision on incorrect information (wrong product output)‏
the customer bases an environmental decision on incorrect information
the customer loses information
the customer loses time/production because the product doesn’t work
the customer loses confidence in the product
9/22/2018

20. Examples of product risks - 2
It may be possible that …
the rebate calculation is incorrect
the monthly report covers the wrong period
users cannot generate invoices
the nightly update is incomplete when the first user wants to use the system in the morning
the system is too slow when a number of users use it simultaneously
a website contains spelling mistakes

21. Risks are found where expectations are in danger of not being met
Risk and expectations
Risks are found where expectations are in danger of not being met
Stakeholders have expectations of the project:
observance of time, resources, and budget,
Stakeholders have expectations of the product:
optimally expectations are expressed as requirements
requirements are transformed into design
requirements are fulfilled in the code (and other sub-products)‏
Meeting expectations is the key to success!

22. Techniques for identifying risks
Personal experience
Checklists and/or risk templates
Risk workshops, e.g. FMEA
Brainstorming
Expert interviews
Independent risk evaluations
Hazard analysis / danger analysis
Fault tree analysis
The methods can be combined

23. Experiences aren’t always reliable
Danger of ‘too late’ experience gathering :
we forget
we only remember successes
we only remember failures
we exaggerate
we downplay / understate
we confuse things
we simply remember wrongly

24. Check lists
Check lists are:
lists of typical risks in context
a valuable asset in an organisation
formed and maintained by experience – knowledge gained from previous projects
structured in relation to the requirement or design specification
good points of departure for risk workshops and brainstorms

25. Risk workshops
Workshops are an effective way to identify risks
Involve as many stakeholders as possible
Ensure that there is a neutral facilitator present capable of leading the discussion
Start by asking the participants to write down their opinion of where the finished system will fail
Permit discussions (not conflicts)
Check and update checklists
All concerns should be documented and analysed

26. A couple of other rules:
Brainstorms
Brainstorms are informal meetings whose purpose is to identify as many possible risks as possible
Rule no. 1: No commenting on ideas allowed even the stupidest or most peculiar idea can provide inspiration – and may generate valuable ideas
A couple of other rules:
A brainstorm must have a facilitator
All ideas must be registered, e.g. on a board
All ideas must be analysed and identified as a risk or removed from the register

27. Expert Interviews Interviews are not just conversations
Find the right people
Write an interview guide to obtain the right information
Follow the interview plan
Permit improvisation
Allow all interviewees to have their say
Avoid closed-ended questions, except where intended
Avoid ‘why?’ instead ask ‘what?’
Take plenty of notes
Interviews can be used to supplement workshops

28. A risk register can work as a checklist It may contain:
Using risk templates
A risk register can work as a checklist
It may contain:
A risk number or title (id)‏
A risk description
Probability
Effect
Level
Mitigating actions
Priority
Dependencies and assumptions
Tables and spread sheets are useful tools for creating risk registers ID
Risk description P. E.
Level
Action
Prio
Comments

29. Risk identification
Exercise

30. Risk-Based Testing Risk Management – Risk Analysis Introduction
Risk Management – Risk Identification
Risk Management – Risk Analysis
Risk Management – Risk Mitigation

31. Risk analysis
Risk analysis is the study of identified risks
puts identified risks in perspective
classifies risks in categories and types
should be performed by the relevant stakeholders
can be more or less stringent
can be more or less objective
can require the participation of experts
determines risk levels for each risk element
is used to determine when and how to test

32. Everything is relative
Risk analysis is mainly based on perceptions
there are very few reliable measurements
perceptions are subjective
different people have different ‘pain thresholds’
different professions have different viewpoints
Strengthen communication and understanding between stakeholders – compromise where necessary
“If you do that, I never want to see you again!!!”
“Is that a promise or a threat?”

33. Different professions consider risk in different ways: Project manager
Various viewpoints
Different professions consider risk in different ways:
Project manager
often under pressure, used to compromising
Developer
proud of the job he’s done and knows how he did it
Tester
often pessimistic, remembers previous experiences
End user
normally has a high error tolerance, is used to finding ways of working round problems

34. Risks have two dimensions
A risk is affected by:
the probability of the problem occurring - technical risk
the effect, in the event that the problem occurs - business risk
Risk level = probability x effect
No probability or no effect = 0 risk
Complete certainty = not a risk but a problem

35. Define independent scales for probability and effect Scales can be:
Measuring risks
Define independent scales for probability and effect
Scales can be:
quantitative – actual figures
may be difficult to use
may be misleading
qualitative – arbitrary scales
are easier to use
less rigorous
just as valuable as quantitative

36. Probability scales
Quantitative probability, typically
0 - 1
0% - 100%
Qualitative scales
not likely, likely, very likely
low, medium, high
1, 2, 3, 4, 5, 6
Minimum 3 intervals – but: an even number of intervals makes analysts think more

37. Evaluation of probability
Example of a qualitative scale
NEVER let the effect influence your evaluation of probability! 1
Highly improbable
10 - 1 2
Improbable 3
Less than fifty-fifty 4
More than fifty-fifty 5
Probable 6
Highly probable
99 – 91
Points
Description
Prob. %
Inspired by: Risk-Based E-Business Testing, Paul Gerrard

38. Effect scales
Actual cost
measured in an agreed currency ($, €, DKK.)‏
open scale
can be difficult to calculate
can be an eye-opener
Qualitative evaluation
bad, worse, worst
1, 2, 3, 4, 5, 6

39. Actual costs can include - 1
the time spend determining that something is wrong
time spent reporting the incident to support
the time spent by support in investigating and attempting to alleviate the problem
double or extra work which the user has to perform
production losses, caused by the system malfunctioning
time to escalate to the next level of support
time used at the next support level

40. Actual costs can include - 2
time spent analysing the deviation and deciding how to deal with it
time spent finding the defect / defects
time spent implementing changes and testing all affected objects
time spent re-testing and regression testing
time spent installing the new version
time spent updating whatever has been done during the malfunction of the system

41. NEVER let the probability affect your evaluation of the effect!
Evaluating effects
NEVER let the probability affect your evaluation of the effect!
Example of a qualitative scale 1
No noticeable effect
Minimal 2
Operations will be somewhat affected
Low 3
Goals will be somewhat affected
Below average 4
Goals will be significantly affected
Over average 5
Goals will be undermined
High 6
Goals cannot be achieved
Critical
Points
Description
Effect
Inspired by: Risk-Based E-Business Testing, Paul Gerrard 41

42. Risk matrix
Effect 6 5 4 3 2 1
Probability

43. Final risk level
Risk level = effect x probability
Used to prioritise risks
Used to steer preventative or mitigating measures
Use spread sheets to ease maintenance

44. Risk level is an estimate
It’s impossible to predict things – especially in the future!
All risk analysis entails a level of uncertainty
Use test results as input for ongoing risk analysis
more defects that anticipated => higher probability => higher risk level
fewer defects than anticipated = > lower probability => lower risk level
Testing should reflect the risk level

45. Risk analysis
Exercise

46. Risk-Based Testing Risk Management – Risk Mitigation Introduction
Risk Management – Risk Identification
Risk Management – Risk Analysis
Risk Management – Risk Mitigation

47. Risk mitigation is the response to the analysed risks
what should we do?
when should we do it?
how do we do it?
STOP
Mitigate: to make less serious or damaging

48. Plan contingency activities
What can we do
Nothing
when the advantage of waiting is greater than the cost of acting
Share the damage
share the risk with the supplier, customer, or other party
Plan contingency activities
contingency plans in the event that risks are realised
Initiate preventative measures
this is where testing can come in

49. Project risks threaten the completion of the (test) project people
availability, skills, knowledge, personalities
time
money
surroundings
processes, tools, hardware
external interfaces
customer / supplier relations

50. Reducing project risks
Test project risks must be managed in close cooperation with the project manager
Start early:
follow the project from the start – and all the way
planning
progress
product quality
change frequency
begin to prepare the project test plan and level test plans at as early a stage as possible

51. Typical test project bottlenecks
Common examples of test process risks:
lack of understanding of the necessity of testing
lack of processes, templates, standards etc.
lack of resources
poor product quality – not least requirements
multiple late changes to the test basis
problems with tools
delayed and/or unsuitable test environment

52. functional requirements non-functional requirements design programming
Product risks
Product risks threaten the product’s quality in relation to the customers needs and requirements
functional requirements
non-functional requirements
design
programming
Here are the defects
Testing also involves risks
Testing can reduce risks

53. Testing is a risk reducing activity
The majority of product risks relate to the presence of defects.
Testing (and defect correction) reduce the level of risk
a test aims to identify defects by making the product fail – before it reaches the customer
defects identified during testing can be corrected - before the system reaches the customer
in areas in which the test does not uncover failures we know that the product works

54. strategy and planning based on risk analysis
Identified and analysed risks can be used to:
determine the levels and types of test to be performed
determine test formality level
prioritise test activities
distribute the test time
Document decisions using the relevant test strategy and/or test plan

55. Determine tests from determined risks
high probability of defects => static test, component test and functional system test
risks relating to user interfaces => usability evaluation and test
risks relating to performance => performance test
risks relating to external interfaces => system integration test

56. Particular formality for particular risks
extensive risk => use specific test case design techniques and high degrees of independence and strict test exit criteria and strict procedures for re-tests and regression testing
small risk => freer choice of test case design techniques and/or lower degree of independence and/or freer test exit criteria and/or freer procedures for re-testing and regression testing

57. Priority and test time based on risk level
Use the final level for prioritization
Use final level distribution to distribute test time TP 1 3 2 4 % 44 17 26 13 46

58. Two approaches
depth first tests performed in risk order from the highest to the lowest risk level
width first test are carried out across all identified risks such that each risk is covered at least once
But what if we can’t make it?

59. Test objective
Gaining information to ensure the quality of
the product (find failures and follow development)‏
decisions - the better the knowledge available, the better decisions can be made
processes (find root causes and weak quality assurance processes)‏
Information is invisible – unless we make it visible!

60. When time gets scarce
Renewed test planning should take place on the basis of an up-to-date risk analysis
The responsible manager may choose to:
increase test time, where possible, and prioritise and distribute the remaining time
release the product with unresolved risks - i.e. transfer the risks to subsequent stakeholders

61. If time is abundant
Renewed test planning should take place on the basis of an up-to-date risk analysis
Use the results from the tests already performed:
where were large/small numbers of defects found?
where have defects been corrected?
what type of defects were found?
where is test-coverage low?

62. Risk mitigation
Exercise