1. Copyright © 2009 PMI RiskSIGNovember 5-6, 2009 “Project Risk Management – An International Perspective” RiskSIG - Advancing the State of the Art A collaboration of the PMI, Rome Italy Chapter and the RiskSIG

2. Slide 2November 5-6, 2009 Copyright © 2009 PMI RiskSIG A framework for measuring operational risks Sabrina Grimaldi Carlo Rafele Politecnico di Torino Department of System Production and Business Economics

3. Slide 3November 5-6, 2009 Copyright © 2009 PMI RiskSIG Operational Risk “The risk in loss resulting from inadequate of failed internal processes, people and systems or from external events” [Basel Commitee on Banking Supervision]

4. Slide 4November 5-6, 2009 Copyright © 2009 PMI RiskSIG Framework aim The proposed methodology is consequently aimed at supporting analysts in estimating a project's exposure to operational risk, in all those situations characterized by a lack of data, by incomplete data or by low- quality data.

5. Slide 5November 5-6, 2009 Copyright © 2009 PMI RiskSIG Uncertainty and Risk Risk is measurable uncertainty Uncertainty is unmeasurable risk Uncertainty is described as epistemic Risk is described as aleatoric Uncertainty relates to a lack of knowledge about possible outcomes of an event, including both their nature and associated probabilities. Risk is an event where the set of possible outcomes is known, and the probability of obtaining each outcome can be measured or estimated, but the precise outcome in any particular instance is not known in advance. [D. Hillson]

6. Slide 6November 5-6, 2009 Copyright © 2009 PMI RiskSIG Uncertainty and Risk Risk is: Uncertainty is: Ignorance “I don’t know what I don’t know” Amnesia “I don’t know what I know” Caution “I know what I don’t know” [D. Hillson]

7. Slide 7November 5-6, 2009 Copyright © 2009 PMI RiskSIG Project Risk Management Process 1) Risk Analysis 1A) Identification 1B) Assessment (qualitative and/or quantitative) 2) Risk Response 2A) Response Development 2B) Response Control

8. Slide 8November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Risk Assessment Qualitative Risk Prioritizing risks for subsequent further analysis or action by assessing and combining their probability of occurence and impact. Quantitative Risk Numerically analyzing the effect on overall project objectives of identified risks. [PMBOK 3° ed.]

9. Slide 9November 5-6, 2009 Copyright © 2009 PMI RiskSIG Project Risk Management Process Complete Information RiskUncertainty Identification phase Assessment phase Planning phase Complete ignorance

10. Slide 10November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Analysis Diverse Information need different methodologies to be processed in order to obtain reliable probability and impact values The risky event assessment phase may take place in different contexts each characterized by a specific informational domain, which needs to be menaged by specific methodologies.

11. Slide 11November 5-6, 2009 Copyright © 2009 PMI RiskSIG One of the most effective methodology for risk identification is represented by the so called Risk Breakdown Matrix (RBM): bi-dimensional matrix, one made of risk sources and the other of corporate processes. RBM perfectly re-enact risky event manifestation process. Risk Analysis

12. Slide 12November 5-6, 2009 Copyright © 2009 PMI RiskSIG risky event manifestation General model of risky event manifestation Risk Factor INT. EXT. Project Risk Risk effects Impact Probability X% Cause/SourcesManifestation OriginConsequenceTransition Effect analysis

13. Slide 13November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Identification: WBS Operational risk origin coincides with all the activities taking place in the assessed company Risk identification starts with corporate process analysis. Work Breakdown Structure

14. Slide 14November 5-6, 2009 Copyright © 2009 PMI RiskSIG Work Breakdown Structure

15. Slide 15November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Identification : RBS Corporate processes to some extent provide a favorable environment for risky events, but they are not their primary source. A source, that is the element originating the risky event for the company, needs in fact to be associated to any “favorable environment”. As far as operational risk is concerned, four main sources are to be considered: People, technology, processes, external sources Risk Breakdown Structure

16. Slide 16November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Structure The RBS is “A source-oriented grouping of project risks that organises and defines the total risk exposure of the project. Each descending level represents an increasingly detailed definition of sources of risk to the project” (Hillson,2002-1). The RBS is therefore a hierarchical structure of potential risk sources, which can be an invaluable aid to understanding the risks faced by the project, acting as a framework to structure and guide the risk management process

17. Slide 17November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Structure Neutral indications of risk source

18. Slide 18November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Structure RBS content is absolutely neutral, in the sense that economic stability of the supplier and not the possibility of his failure is considered. The source is a system state; the event acts as an output of such a state, which is characterized by a certain probability to happen. Starting from the internal system characteristics a series of events characterized by positive or negative impacts on company output may take place. The taking place possibility is measured by probability.

19. Slide 19November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Matrix

20. Slide 20November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Matrix A RBM cell does not necessarily need to contain a specific event. As a matter of fact, there could be processes where a source does not cause any risky event. The RBM is important since it helps distinguish processes from sources. Especially when dealing with operational risks, the analyst could be induced to consider processes as sources of risk, thus making relevant mistakes.

21. Slide 21November 5-6, 2009 Copyright © 2009 PMI RiskSIG Event analysis Risky events could be basically divided into two main categories: 2. Single events 1. Repetitive events Events related to the development of new products or projects Events related to manufacturing processes

22. Slide 22November 5-6, 2009 Copyright © 2009 PMI RiskSIG The events categories in the Project Risk Management Process Complete Information RiskUncertainty Identification phase Assessment phase Planning phase Complete ignorance Single events Repetitive events

23. Slide 23November 5-6, 2009 Copyright © 2009 PMI RiskSIG Repetitive event analysis The outcomes of a company may be described as functions of the performances of its main processes as monitored over time. E.g. processes in a company manufacturing mechanical components: machines, tools and products will probably change in the long term, but the company has a specific production capacity, given processes and a fixed set of products in the short term. Repetitive Events Classical models: Probability Analysis, Monte Carlo Simulation quantitative approach

24. Slide 24November 5-6, 2009 Copyright © 2009 PMI RiskSIG Single event analysis Order projects, such as the construction of a bridge, a dam, or a manufacturing plant… The main characteristic of these projects is their uniqueness. As a matter of fact, even if it is managed by standardized procedures, a order project is unique because of the product to be developed (and a project team has been formed for this purpose), the people working on it, third party stakeholders, execution times, the assigned financial budget, the strategic role of the final product, and its market. Single events Conceptual models: Fuzzy Analysis qualitative approach

25. Slide 25November 5-6, 2009 Copyright © 2009 PMI RiskSIG Single event analysis Risks related to the performances of a project for new product development processes are quantified by means of expert judgments. Experts are people in the company having a good experience and knowledge. Their evaluation could be integrated with available data about past projects. For instance, there is no doubt that the relationship between the product development delay (a popular consequence of many risks) and the company outcomes is very well known to experts, even if it is not possible to express it in a mathematical form. Usually, this knowledge comes from both past experiences and the results of studies performed on the topic by specialized organizations.

26. Slide 26November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Assessment: qualitative approach experts The qualitative approach should be used in all those situations characterized by a complete lack of numerical data, where the people we have called experts are the only source of information available for the risk analyses Qualitative Model Supported by: Elicitation Methodology Fuzzy Function FMEA/FMECA

27. Slide 27November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach The qualitative approach may be divided into two phases: 1.Experts are interviewed one by one (elicitation methodology); 2.Experts are brought together to discuss the results expressed during their respective interviews (individual judgment aggregation).

28. Slide 28November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: elicitation methodology (phase 1) Using the elicitation methodology, analyst (project manager, risk manager, etc.) sets a process aiming at eliciting from experts’ experience as much information as possible, for the purpose of obtaining precise estimates about the impact and the occurrence probability of an event. Expert knowledge elicitation is: “a formal process to obtain information or answers to specific questions about problems that need to satisfy certain analytical goals”; “a process transforming people knowledge and beliefs about one or more uncertain quantities into appropriate forms”.

29. Slide 29November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: elicitation process The elicitation process is performed according to the following steps: 1.Preliminary information: event description, field where the risk comes from, expert’s job, expert’s objective characteristics,... 2.Explaining the reasons for the analysis: the analyst explains to the expert how the analysis will be performed 3.Setting the analysis: a first range for all the possible values of both occurrence times and impacts is defined 4.Conditioning: the analyst helps the expert look at the problem from different points of view 5.Knowledge investigation: the analyst tries to understand the expert’s degree of specific knowledge 6.Defining the probability distributions for the risky event occurrence and impact 7.Knowledge verification: the analyst shares the obtained results with the expert.

30. Slide 30November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: Failure Mode and Effects (Criticality) Analysis FMEA/FMECA An analysis technique used in high- risk organizations to identify and assess failure mode in systems/processes and work out response strategies.

31. Slide 31November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: FMEA/FMECA STEPS TO CHOOSE THE PROCESS TO ANALYZE THE PROCESS TO IDENTIFY THE POSSIBLE “FAILURE MODES” TO DETERMINE THE CONSEQUENCES TO INDIVIDUALIZE THE CAUSES TO DETERMINE THE CRITICALITY' INDEX TO DEFINE REDUCTION ACTIONS TO ESTIMATE EFFECTIVENESS REDUCTION ACTIONS

32. Slide 32November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: FMEA/FMECA TO CHOOSE THE PROCESS TO ANALYZE THE PROCESS ABS

33. Slide 33November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: FMEA RBS TO IDENTIFY THE “FAILURE MODES” TO DETERMINE THE CONSEQUENCES TO INDIVIDUALIZE THE CAUSES TO DEFINE REDUCTION ACTIONS CARD FMEA

34. Slide 34November 5-6, 2009 Copyright © 2009 PMI RiskSIG RBS CARD FMECA Unique process risks : scheda FMECA TO IDENTIFY THE “FAILURE MODES” TO DETERMINE THE CONSEQUENCES TO INDIVIDUALIZE THE CAUSES TO DEFINE REDUCTION ACTIONS TO DETERMINE THE CRITICALITY INDEX

35. Slide 35November 5-6, 2009 Copyright © 2009 PMI RiskSIG FROM RISK TO WASTE A process out of control that causes risks, it is also an inefficient sequence of activities. This a sure cause of waste. Thus in a similar could be useful to analyze both the efficiency as the efficacy side.

36. Slide 36November 5-6, 2009 Copyright © 2009 PMI RiskSIG TOYOTA 7 WASTE: Transport : Movement of materials is a waste. Minimize the amount of movement by arranging processes in close proximity to each other. Inventory : Too little inventory can lose sales, too much inventory can hide problems. Motion : Remove unnecessary motion of the operations and improve the ergonomics of the workplace. Waiting : Minimize waiting time and maximize "value adding" time. Aim for a smooth flow. Overproduction : Always aim to make exactly what the customer orders, just in time, to the correct quality standard. OverProcessing : Use machines which are of an appropriate capacity and capable of achieving the required quality standard. Defects : Reducing the number of defects directly reduces the amount of waste. Aim for zero defects.

37. Slide 37November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: FMEA + WASTE: RBS TO IDENTIFY THE “WASTE” TO DETERMINE THE CONSEQUENCES TO INDIVIDUALIZE THE CAUSES TO DEFINE REDUCTION ACTIONS CARD Waste - FMEA

38. Slide 38November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: CRITICALITY INDEX Risk priority numbers = Probability X Impact TO DETERMINE THE CRITICALITY INDEX RBS FUZZY FUNCTION

39. Slide 39November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: fuzzy functions Traditional elicitation methods aim at obtaining a probability distribution (for impact and occurrence frequency of risky events). In our analysis we have replaced probability distributions with fuzzy functions for the following reasons: the consistency constraint is released, thus giving a further degree of freedom to experts’ judgment; experts in any company are usually little familiar with the statistical concepts of median, mode, quantile, and variance; fuzzy distributions provide experts with the same graphical support as probability distributions, but without introducing too many constraints and mathematical concepts.

40. Slide 40November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: fuzzy membership function The fuzzy membership function is the analytical tool project manager use to translate experts' knowledge into a formally correct language. EXAMPLE: suppose you want to estimate the width of your colleague’s desk without measuring it, but just interviewing the colleague. This person will give you: a range of possible values, outside which he thinks the real width cannot be; a range of most likely values including, in his opinion, the correct measure of the desk. 40-90 is the range of possible values 60-80 is the range of most likely values

41. Slide 41November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: standard ranges Since the purpose of the first phase of the approach (individual interviews) is identifying the less significant events from a risk point of view, analysts need to provide experts with evaluation scales. The proposed procedure suggests to define standard ranges basing on the experience gained by analysts in the studied sector over time. Class label Impact 1.Negligible or very low< 0,5% 2.Lowfrom 0,5 to 2% 3.Mediumfrom 2 to 5% 4.Highfrom 5 to 10% 5.Catastrophic or very high> 10%

42. Slide 42November 5-6, 2009 Copyright © 2009 PMI RiskSIG The following step requires to translate range boundaries from percentage to absolute values, strictly depending on the company dimension. Class label Impact 1. Negligible or very low< 5.000€ 2. Lowfrom 5.000 to 20.000€ 3. Mediumfrom 20.000 to 50.000€ 4. Highfrom 50.000 to 100.000€ 5. Catastrophic or very high> 100.000€ In order to agree on a reference metric on which basing the analysis, it is highly recommended to define the extreme values of each range together with the company’s top management Example: company with a turnover of about 1.000.000 Euros Qualitative approach: standard ranges

43. Slide 43November 5-6, 2009 Copyright © 2009 PMI RiskSIG In order to determine the occurrence probability, a time horizon is set, for example including the business cycles related to the company’s processes. After that, the ranges defining the number of times an event may occur in the given time horizon are identified. Class label Probability 1.Negligible or very low0 - 0.2 2. Low0.2 - 0.4 3. Medium0.4 - 0.6 4. High0.6 - 0.8 5. Very high0.8 - 1 Example: time horizon of 18 months Qualitative approach: standard ranges

44. Slide 44November 5-6, 2009 Copyright © 2009 PMI RiskSIG After performing individual interviews with experts, analyst should have only two representations: one about the event impact, and the other about its occurrence probability. The numbers 1, 2, 3, 4, and 5 on the x-axis correspond to the five classes of impact and frequency defined before. Qualitative approach: range representation IMPACTPROBABILITY

45. Slide 45November 5-6, 2009 Copyright © 2009 PMI RiskSIG The judgment should result from the contributions of all the experts, and not be a pure mathematical combination of individual judgments. IMPACTPROBABILITY 1° expert 2° expert IMPACTPROBABILITY Qualitative approach: combining individual judgments (phase 1)

46. Slide 46November 5-6, 2009 Copyright © 2009 PMI RiskSIG Qualitative approach: combining individual judgments (phase 2) The following step is aggregating individual judgments, in order to come out with a univocal evaluation of the probability and impact for each identified risky event. The main aim of the group part of the analysis is basically narrowing the range of possible values, and moving from a distribution very similar to a trapezoidal to a uniform one. IMPACTPROBABILITY

47. Slide 47November 5-6, 2009 Copyright © 2009 PMI RiskSIG Graphical representation of results IMPACT PROBABILITY IMPACTPROBABILITY

48. Slide 48November 5-6, 2009 Copyright © 2009 PMI RiskSIG Alfa Company Case 1.Available materials analysis 2.Risk Breakdown Matrix methodology application Alfa core process identification Potential risky causes analysis Risky events causes and process intersection Risky events identification 3.Identified risks must be divided into 2 big groups: Unique process linked risks Repetitive process linked risks 4.Unique process linked risk analysis: qualitative method is based on expert judgment

49. Slide 49November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risk Breakdown Matrix

50. Slide 50November 5-6, 2009 Copyright © 2009 PMI RiskSIG Risky events (1) A specific consequence is associated with risky events identified in RBM cells. EVENTCONSEQUENCE 1 Non-application of the existing development process Missing deadlines 2 Insufficient collaboration of suppliers in partnership/codesign relations Product launch delays 3 Insufficient information from the marketing sector Lower innovation and quality degree towards competitors 4 Lack of resources in product development process Product launch delays 5Sensitive information diffusionInformation appropriation by competitors 6Little reliability of funded dealersFailed payback from the marketing network 7Poor quality of supplied componentsProduction delays and non-quality cost 8Failed optimization of productivityUnsatisfied demand or underuse 9Lack of Human and material ResourcesInadequate production (qualitative and quantitative) 10Lack of materials on the line edgeLine breakdown

51. Slide 51November 5-6, 2009 Copyright © 2009 PMI RiskSIG Unique process risks: a qualitative approach Event 3: “not sufficient information from the marketing sector", imply a “lower innovation and quality degree towards competitor". Events with lower values are taken out The search for quantitative measures describing the situation is both utopist and having negative implications on reality. Step 1 Single interviews Step 1 Single interviews Qualitative approach Step 2 Group interaction

52. Slide 52November 5-6, 2009 Copyright © 2009 PMI RiskSIG Unique process risks : process card PROCESSESACTIVITY MARKETING Resources Commercial staff Input Market Information, product costs Description To compare market demand with company offer Time 1 week; after market survey, before product selling Control Marketing manager checks the analysis Tools Market survey, company cost database Output Product price

53. Slide 53November 5-6, 2009 Copyright © 2009 PMI RiskSIG Unique process risks : scheda FMECA Cod. Sub- processes Failure Modes CausesEffectsExpertCriticality IndexSurvey methodology Failure Modes Actions to Reduce Failure mode Probability (class label) Impact (class label) Pegging selling prices Wrong price determi nation Wrong market inform ation Not correct market target Market ing manag er 2 - 4 Benchmarking on retail network Two indipendent market analysis Pegging selling prices Wrong price determi nation Mistak es in cost evaluat ion Not competit ive product Compa ny control ler 2 - 42 - 3Benchmarking on retail network Deeper industrial cost reporting

54. Slide 54November 5-6, 2009 Copyright © 2009 PMI RiskSIG Unique process risks : FMEA + WASTE: Cod. Subproc Waste Mode CausesEffectsSurvey methodology Waste Modes Actions to Reduce Waste mode Pegging selling prices Excess of data collecting on useless sectors Market analysis not well focused Incomplete informatio n for price definition Definition of a market analysis standard Deeper segmentation of the market

55. Slide 55November 5-6, 2009 Copyright © 2009 PMI RiskSIG Unique process risks: a qualitative approach (step 1) The results collected in the interviews to 2 experts about "not sufficient information from the marketing sector " are represented in the following fuzzy function. 1° expert 2° expert IMPACTPROBABILITY IMPACTPROBABILITY

56. Slide 56November 5-6, 2009 Copyright © 2009 PMI RiskSIG On the basis of the 2 experts’ judgments we may notice a partial discrepancy: different value ranges about event manifestation probability and money impact. Should we chosen a one step elicitation process, at that point we have to combine mathematically the 2 experts’ functions, by keeping in mind the reliability degree associated to each expert. Unique process risks: a qualitative approach (step 1)

57. Slide 57November 5-6, 2009 Copyright © 2009 PMI RiskSIG In the graph below impact evaluations, combined the 2 expert’ judgments. The area colored in blue displays the result of the intersection only. Unique process risks: a qualitative approach (step 1)IMPACT

58. Slide 58November 5-6, 2009 Copyright © 2009 PMI RiskSIG Group session effectiveness is well represented by the graph, reflecting the uncertainty reduction initially associated to individual judgments. A lower uncertainty means lower complexity: the analyst at this point has at his disposal a range of possible values both in terms of probability and impact. Instead, the group interaction path is different: the goal is an exchange of views and information sharing among experts, leading each expert to formulate a judgment which is not exactly in harmony with that expressed by the others. IMPACTPROBABILITY Unique process risks: a qualitative approach (step 2) 0.20.40.60.8

59. Slide 59November 5-6, 2009 Copyright © 2009 PMI RiskSIG Event 5 :“Sensitive information diffusion” imply a “Information appropriation by competitors” 1° expert 2° expert Unique process risks: Fuzzy Function IMPACTPROBABILITY IMPACTPROBABILITY

60. Slide 60November 5-6, 2009 Copyright © 2009 PMI RiskSIG In this case both experts gave low values about the manifestation probability and impact. Following a Pareto analysis after all preliminary individual evaluations, the event would inevitably fall within zone C with the subsequent exclusion from the group analysis. Such an exclusion is essentially linked to external and internal resources cost: decreasing the number of events to be assessed following individual interviews is fundamental to gather the limited resources and employ them for the most important events. Unique process risks: a qualitative approach

61. Slide 61November 5-6, 2009 Copyright © 2009 PMI RiskSIG In this case the agreement of the 2 judgments is carried out by the project manager through a mathematical combination, where all the experts’ judgments are equally considered. See graphs below. Unique process risks: a qualitative approach IMPACTPROBABILITY 0.20.40.60.8

62. Slide 62November 5-6, 2009 Copyright © 2009 PMI RiskSIG Event 2 : “insufficient collaboration of suppliers in partnership/codesign relations". Information provided by expert are characterized by a very high precision level. Unlike the 2 previous 2 risky events, no group aggregation follows the individual interview. Unique process risks: a qualitative approach IMPACTPROBABILITY 0.20.40.60.8

63. Slide 63November 5-6, 2009 Copyright © 2009 PMI RiskSIG Following a qualitative analysis of the risky events we find risk map. The table below summarizes the results obtained from the 3 events considered. IDEventImpactProbability 3Not sufficient information from the marketing sector 25.000 – 40.000 €0.3 – 0.4 5Sensitive information diffusion5.000 – 25.000€0.1 - 0.2 2Insufficient collaboration of suppliers in partnership/codesign relations 83.000 – 85.000 €0.2 Unique process risks

64. Slide 64November 5-6, 2009 Copyright © 2009 PMI RiskSIG This risk map is a bidimensional graph with the manifestation probability on the x axis and impact on the y axis; each event is represented by a segment. In this way we may have a direct picture of assessment- associate uncertainty corresponding to the segment length. Unique process risks [C. Chapman ]

65. Slide 65November 5-6, 2009 Copyright © 2009 PMI RiskSIG The described methodology allows to analyze the operational risks characterized by incomplete or by low-quality data. This methodology proposes framework integrating critical process analysis with a careful understanding and managing of sources of risks and waste. Following the proposed approach, decision makers can identify, implement and review strategies driving them towards efficiency The output is a risk map that clearly shows relative importance among the risks and clarifies interventions priorities. Conclusion

66. Copyright © 2009 PMI RiskSIGNovember 5-6, 2009 “Project Risk Management – An International Perspective” RiskSIG - Advancing the State of the Art A collaboration of the PMI, Rome Italy Chapter and the RiskSIG