1. Assessing & Measuring Operational Risk Why COSO is Inappropriate ISDA PRMIA London, United Kingdom January 18, 2005 Ali Samad-Khan President, OpRisk Advisory LLC ali.samad-khan@opriskadvisory.com www.opriskadvisory.com

2. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Agenda IIntroduction IIDefinition and Categorization IIICOSO Based Risk Assessment IVIntegrated Risk Measurement and Management VAlternative Approaches to Risk Assessment VIControl Assessment VIISummary & Conclusions

3. INTRODUCTION

4. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. When considering risk and control assessment, what are the priorities? Establish a disciplined process. It’s the process that matters, the results are less important A good process lays the foundation for a good risk management culture Establish a process that will produce the most reliable results. The results are more important If it is clear to end users that the results are fictitious then the entire risk management program will be discredited; the operational risk program will be seen to be adding little value Demonstrate practical value and the program will be a success and subsequently create the right culture

5. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. RISKSCONTROLS What type of risks do I face? Which are the largest risks? How well are these risks being managed? Manage Controls through Cost Benefit Analysis Consider introducing a disciplined process that produces accurate results which facilitate educated decisions making.

6. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. To ensure the goal is practical, one needs to express it in the context of a business problem Consider two risks: Unauthorized Trading and Money Transfer Past Audits reveal that both risks are under-controlled To address Unauthorized Trading risk one must improve segregation of duties and audit frequency. (Solution: hire four new staff; cost = $600,000 per year) To address Money Transfer risk one must improve the system (Solutions: buy new system; cost = $5 million) You have $4 million in your budget. Where do you invest your money?

7. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. FOUNDATION Risk strategy, tolerance Roles and responsibilities Policies and procedures Risk definition and categorization FOUNDATION Risk strategy, tolerance Roles and responsibilities Policies and procedures Risk definition and categorization INFORMATION Expected Loss – how much do I lose on average Unexpected Loss – how much I could reasonably expect to lose in a bad year Control Scores – how good are the controls I have in place INFORMATION Expected Loss – how much do I lose on average Unexpected Loss – how much I could reasonably expect to lose in a bad year Control Scores – how good are the controls I have in place DATA Loss data collection Risk indicator data collection Control self- assessment Risk assessment and analysis Automatic notification Follow up action reports DATA Loss data collection Risk indicator data collection Control self- assessment Risk assessment and analysis Automatic notification Follow up action reports MANAGEMENT Awareness of real exposures Knowledge of controls quality Cost benefit analysis Improved risk mitigation and transfer strategy MANAGEMENT Awareness of real exposures Knowledge of controls quality Cost benefit analysis Improved risk mitigation and transfer strategy Management & Control Quality Economic Profit Effectively managing operational risk requires a foundation designed to turn raw operational risk data into information that supports managerial decision making

8. DEFINITION AND CATEGORIZATION

9. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. What does operational risk include? Transaction Inadequate Supervision Reputation Insufficient Training Compliance Poor Management Execution Information Relationship People Legal/Regulatory Fixed Cost Structures Settlement Key Man Theft Fraud Fiduciary Customer Business Interruption Technological Lack of Resources Criminal Rogue Trader Physical Assets Business Strategic

10. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Insufficient training CAUSESEVENTSCONSEQUENCES Lack of management supervision Lack of management supervision Inadequate auditing procedures Inadequate auditing procedures Inadequate security measures Inadequate security measures Poor HR policies Poor HR policies Poor systems design Poor systems design Inadequate segregation of duties Inadequate segregation of duties External Fraud External Fraud Employment Practices & Workplace Safety Employment Practices & Workplace Safety Clients, Products & Business Practices Clients, Products & Business Practices Damage to Physical Assets Damage to Physical Assets Business Disruption & System Failures Business Disruption & System Failures Execution, Delivery & Process Management Execution, Delivery & Process Management Internal Fraud Internal Fraud Regulatory, Compliance & Taxation Penalties Regulatory, Compliance & Taxation Penalties Restitution Loss of Recourse Reputation Business Interruption EFFECTS Monetary Losses OTHER IMPACTS Forgone Income Write-down Loss or Damage to Assets Loss or Damage to Assets Legal Liability The universe of operational risks spans causes, events and consequences

11. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Placing loss data within a Business Line/Risk matrix helps reveal the risk profile of each business

12. COSO BASED RISK ASSESSMENT

13. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Risk can also be assessed using a likelihood-impact approach. This approach has been well documented by the Committee of Sponsoring Organizations of the Treadway Commission (COSO). Source: COSO

14. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. COSO Likelihood x Impact = Risk The COSO view of risk assessment is based on the likelihood and impact of a specific type of event; the output is probability weighted impact. The high risk area is in the top right corner of the matrix. COSO High (3) Med (2) Low (1) LIKELIHOOD Low (1) Med (2) High (3) IMPACT 9 6 3 6 4 2 3 2 1

15. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Under the risk management industry approach, the high risk area is the bottom right cell in the matrix. Risk Management Industry COSO High (3) Med (2) Low (1) LIKELIHOOD Low (1) Med (2) High (3) IMPACT n/a

16. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Risk Management Industry COSO When compared, there are significant differences …. 9 6 3 6 4 2 3 2 1 COSO Low (1) Med (2) High (3) Impact High (3) Med (2) Low (1) Likelihood Real Risks Phantom Risks COSO n/a High (3) Med (2) Low (1) Likelihood Low (1) Med (2) High (3) Impact

17. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Likelihood x Impact = Risk Under the COSO approach one calculates risk through likelihood-impact analysis Risk 1 : 10% x $10,000 = $1,000

18. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Likelihood x Impact = Risk Likelihood-impact analysis can yield more than one result Risk 1 : 10% x $10,000 = $1,000 Risk 2 : 1% x $50,000 = $ 500

19. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Likelihood x Impact = Risk Using likelihood-impact analysis one can calculate multiple outcomes Risk 1 : 10% x $10,000 = $1,000 Risk 2 : 1% x $50,000 = $ 500.... Risk 999 : 5% x $25,000 = $1,250 Risk 1000 : 20% x $ 6,000 = $1,200

20. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. The many probability and impact combinations represent a continuum P 10% x $10,000 1% x $50,000 5% x $25,000 20% x $ 6,000 Impact

21. INTEGRATED RISK MEASUREMENT AND MANAGEMENT

22. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Risk is measured using internal and external loss data. The two measures of exposure are the aggregate mean and aggregate Value at Risk (VaR). INDIVIDUAL LOSS EVENTS RISK MATRIX FOR LOSS DATA VAR CALCULATION TOTAL LOSS DISTRIBUTION 74,712,345 74,603,709 74,457,745 74,345,957 74,344,576 167,245 142,456 123,345 113,342 94,458 LOSS DISTRIBUTIONS Frequency of events Severity of loss Annual Aggregate Loss ($) Mean 99th Percentile VaR Calculator e.g., Monte Carlo Simulation Engine VaR Calculator e.g., Monte Carlo Simulation Engine 43210 P P Risk

23. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Risk is measured using internal and external loss data. The two measures of exposure are the aggregate mean and aggregate Value at Risk (VaR). INDIVIDUAL LOSS EVENTS RISK MATRIX FOR LOSS DATA VAR CALCULATION TOTAL LOSS DISTRIBUTION 74,712,345 74,603,709 74,457,745 74,345,957 74,344,576 167,245 142,456 123,345 113,342 94,458 LOSS DISTRIBUTIONS Frequency of events Severity of loss Annual Aggregate Loss ($) Mean 99th Percentile VaR Calculator e.g., Monte Carlo Simulation Engine VaR Calculator e.g., Monte Carlo Simulation Engine 43210 P P What is the impact of the tail on the mean?

24. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. 0 Current score VAR CONTROL ASSESSMENT/INDICATOR SCORE Adjustment for Quality of Current Control Environment CAPITAL 190 100 210 Previous score 50 Linking capital to changes in the quality of internal controls provides an incentive for desired behavioral change By comparing changes in the control environment one can predict changes in each business’ risk profile

25. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. RISK MATRIX FOR CAPITAL The risk management industry approach can be used to integrate measures of risk and control, which can be used for allocating economic capital

26. ALTERNATE APPROACHES TO RISK ASSESSMENT

27. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. What other approaches can be considered for risk assessment? Directly estimate frequency and severity parameters through expert judgment Estimate frequency and severity distributions using institutional memory Estimate risk (VaR) directly using internal and external loss data and disciplined scenario analysis

28. ALTERNATE APPROACHES TO RISK ASSESSMENT

29. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Even with significant amounts of historical loss data it is virtually impossible to reliably estimate severity parameters.

30. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. It is also very difficult to reliably estimate severity probabilities at different quantiles. Multiple estimates often create internal inconsistency P 1 in 20 years = $25,000 Impact 1 in 100 years = $50,000 1 in 10 years = $10,000 1 in 1 years = $1,000

31. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Disciplined scenario analysis has been found to be moderately reliable and has produced valuable business benefits. The analysis is based on factual, historical (external) loss data Risk magnitude is clearly defined as potential loss at a specified confidence level, such as 99% A 99% level event is defined to mean the second highest loss in one hundred years This is further clarified – put into practical terms – based on loss experiences of ten peer banks; (similar size, similar controls), the second highest loss in the last ten years for the peer group The whole purpose of this analysis is to allow the bank to compare the magnitude of loss at the same probability level: 50 foot tidal wave vs. 100 tidal wave $10 million money transfer loss vs. $100 million sales practices loss

32. CONTROL ASSESSMENT

33. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. We start with the risks, and using loss data identify control weaknesses and their underlying causes Insufficient training LOSSESINDICATOR Lack of management supervision Lack of management supervision Segregation of duties Number of Reconciliation Errors Number of Reconciliation Errors Internal Fraud Number of Customer Complaints Number of Customer Complaints Data Integrity EDPM Staff Training Budget Vacation policy RISKS Data manipulation CONTROL ISSUE CAUSES 74,712,345 32,603,709 457,745 5,345,957 44,576 167,245 142,456 123,345 113,342 94,458 RISKSLOSSESCAUSES Risks are manifested in losses

34. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. The goal is to identify which specific controls ought to be assessed Leakage of Confidential Information Leakage of Confidential Information Special procedure in terms of Business Intelligence Algorithms Special procedure in terms of Business Intelligence Algorithms Internal Fraud – Credit Card Counterfeiting Internal Fraud – Credit Card Counterfeiting Timeliness of information Timeliness of information Special clearance for overseas travelers Special clearance for overseas travelers Access to information Access to information CONTROL ISSUE 2,000 events Losses =$40M RISKSLOSSESCAUSES Risks are manifested in losses

35. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. The next step is to examine the underlying business processes to better understand how well the risks are currently being managed and controlled Process Chart End Applications Received. Data Entry. Completed entry applications forwarded to officers. Credit Information Verification Application processing officers. Officers’ decision: - Approved / Declined / Further information Allocation of Application

36. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. A control assessment scoring process must be relevant, consistent and objective RelevanceThe control issues must be relevant to a business line and risk Answer ChoicesThe answer choices should be consistent WeightingThe control issues must be weighted according to relevance ScaleAll scores must be converted to a consistent scale, e.g., 0 to 100 NormalizationThe process for normalizing scores must be theoretically valid TransparencyThe process must be transparent to allow for buy in and to identify opportunities for improvement ValidationResponses must be validated to avoid “gaming” the system

37. SUMMARY & CONCLUSIONS

38. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Integrated risk measurement and management can produces value, where the results are meaningful Risk assessment is feasible and practical, but must be implemented only after one fully understand the meaning of the word risk and the technical challenges. Control assessment is feasible and practical, but must only be implemented after one understands the how to make a subjective process more objective. Integrated risk and control assessment or measurement promotes educated decision making, which in turn facilitates prudent risk management an can contribute to the creation of a good risk culture.

39. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. COSO was conceived in the early 1990’s – the first attempt at standardizing what was a subjective and inconsistently applied method for identifying, assessing, controlling and managing operational risks However, in the context of modern operational risk management, we have learned: The definition of risk magnitude under COSO is inconsistent with that used in the risk management industry, including the BIS The approach is highly subjective, resource intensive and generates a huge catalog of unmanageable ‘risks’ COSO approach to risk assessment (likelihood - impact analysis) focuses attention on what are likely to be phantom risks, not real risks. It produces both false positives and false negatives. Any prioritization of controls based on this spurious and misleading information may lead to enhancing controls in areas that are already over-controlled, while ignoring areas of control weakness

40. When the answers are unclear… … is it because we are asking the wrong questions?

41. Copyright © 2004, OpRisk Advisory LLC. All rights reserved. Biographical Information Ali Samad-Khan is President of OpRisk Advisory LLC. He has eight years experience in operational risk measurement and management and approximately twenty years of professional experience. His areas of expertise include: establishing an integrated operational risk measurement and management framework, developing policies and procedures, internal loss event database design and implementation; data quality assessment, data sufficiency, risk indicator identification, risk and control self assessment, disciplined scenario analysis, causal/predictive modeling, advanced VaR measurement techniques and economic capital allocation.. Mr. Samad-Khan has advised many of the world’s leading banks on operational risk measurement and management issues. His significant practical experience in this field comes from managing the implementation of ten major operational risk consulting engagements at leading institutions in North America, Europe and Australia. Key elements of the ORA framework and methodology have been adopted by dozens of leading financial institutions worldwide and have also been incorporated into the BIS guidelines. Mr. Samad-Khan has frequently advised the major bank regulatory authorities, including: the Risk Management Group of Basel Committee on Banking Supervision, the Board of Governors of the Federal Reserve System, the Federal Reserve Bank of New York, the Financial Services Authority (UK) and the Australian Prudential Regulatory Authority. He also holds seminars and workshops for the Bank of International Settlements (BIS) and the Institution of International Finance (IIF). Prior to founding OpRisk Advisory, Mr. Samad-Khan was CEO of OpRisk Analytics LLC, which was acquired by SAS in 2003. (From June 2003 to September 2004 Mr. Samad-Khan provided transitional support for the acquisition of OpRisk Analytics, serving as SAS’ Head of Global Operational Risk Strategy.) He has also worked at PricewaterhouseCoopers (PwC) in New York, where for three years he headed the Operational Risk Group within the Financial Risk Management Practice, in the Operational Risk Management Department at Bankers Trust as well as the Federal Reserve Bank of New York and the World Bank. Mr. Samad-Khan holds a B.A. in Quantitative Economics from Stanford University and an M.B.A. in Finance from Yale University. Articles include: “Is the Size of an Operational Loss Related to Firm Size,” with Jimmy Shih and Pat Medapa, Operational Risk, January 2000; “Measuring and Managing Operational Risk,” with David Gittleson, Global Trading, Fourth Quarter, 1998. Working papers include: “How to Categorize Operational Losses – Applying Principals as Opposed to Rules” March 2002 and “Categorization Analysis” January 2003.