Presentation is loading. Please wait.

Presentation is loading. Please wait.

Filename Comptroller of the Currency Administrator of National Banks Managing Model Risk in Retail Scoring Dennis Glennon Credit Risk Analysis Division.

Similar presentations

Presentation on theme: "Filename Comptroller of the Currency Administrator of National Banks Managing Model Risk in Retail Scoring Dennis Glennon Credit Risk Analysis Division."— Presentation transcript:

1 Filename Comptroller of the Currency Administrator of National Banks Managing Model Risk in Retail Scoring Dennis Glennon Credit Risk Analysis Division Office of the Comptroller of the Currency September 28, 2012 The opinions expressed in this paper are those of the authors and do not necessarily reflect those of the Office of the Comptroller of the Currency. All errors are the responsibilities of the authors.

2 Filename 2 Agenda  Introduction to Model Risk  What is it?  Why is it relevant?  Managing Model Risk  Overview of Sound Model Development and Validation Procedures  Emerging Issues Related to Model Risk 2

3 Filename 3 Models Risk: What is it?  Model Risk – Potential for adverse consequences from decisions based on incorrect or misused model outputs  Model errors that produce inaccurate outputs  Model may be used incorrectly or inappropriately (i.e., using a model outside the environment for which it was designed).  Model risk emerges from the process used to develop models for measuring credit risk.  The process introduces a secondary loss exposure beyond that of credit risk alone  e.g., poor underwriting decisions based on erroneous models or overly broad interpretations of model results. 3

4 Filename 4 M odel Risk: What is it?  Credit Risk: The risk to earnings or capital from an obligor's failure to meet the terms of any contract with the bank or otherwise fails to perform as agreed.  A conceptually distinct exposure to loss.  There are many reasons for poor model-based results including:  Poor modeling (i.e., inadequate understanding of the business)  Poor model selection (i.e., overfitting)  Inadequate understanding of model use  Changing conditions in the market 4

5 Filename 5 Managing Model Risk  The goal of model-risk analysis is to isolate the effect of a bank's choice of risk-management strategies from those associated with incorrect or misused model output.  Model Validation is an essential component of a sound model-risk management process.  Validate at time of model development/implementation  Ongoing monitoring  Re-validate 5

6 Filename 6 Model Risk  Model validation can be costly.  However, using unvalidated models to underwrite, price, and/or manage risk is potentially an unsafe and unsound practice.  The best defense against model risk is the implementation of formal, prudent, and comprehensive model-validation procedures. 6

7 Filename 7  Sound modeling practices  In many cases, there are generally accepted methods of building and validating models.  These methods incorporate procedures developed in the finance, statistics, econometrics, and information theory literature.  Although these methods are valid, they may not be appropriate in all applications.  A model selected for its ability to discriminate between high and low risk may perform poorly at predicting the likelihood of default. 7 Model Risk: Sound Modeling Practices

8 Filename 8  Two primary modeling objectives  Classification: The model is used to rank credits by their expected relative performance  Prediction: The model is used to accurately predict the probability of the outcome  Modelers typically have one of these objectives in mind when developing and validating their models 8 Models as Decision Tools

9 Filename 9 9 0 1 y Score (quintiles) y 1 0 20 4080600100 1030507090 0 10 20406080100 30705090 Score (quintiles) 97531 116521 17 4 95 154 3 [0.1] [0.08][0.45][0.44][0.67][0.92] [0.3][0.5][0.7][0.9] [#B / (#G + #B)] [bad rate] obs. bad (B) - y=1obs. good (G) - y=0 Model 2 Model 1 Model Selection: Which model is better?

10 Filename 10  A comparison of models: visual summary Reliable and Accurate Reliable, but not Accurate Models as Decision Tools Odds: 33:1 Bad %: 3.0% Score: 253 Odds: 12.2:1 Bad %: 7.6% Odds: 33:1 Bad %: 3.0%

11 Filename 11 Illustrative Example ln(20/1) = 3.0 bad rate = 5% ln(4/1) = 1.4 bad rate = 20%

12 Filename 12  The model design should reflect how the model will be used.  As such, the choices of:  sample design  modeling technique  validation procedures should reflect the intended purpose for which the model will ultimately be used.  To effectively manage model risk, the right tools must be used. 12 Models as Decision Tools

13 Filename 13  Models are developed for different purposes – i.e., classification or prediction. As such, the choices of:  sample design  modeling technique  validation procedures are driven by the intended purpose for which the model will ultimately be used. 13 Models as Decision Tools

14 Filename 14 M odel Validation  The classification objective is the weaker of two conditions.  There are well-developed methods outlined in the literature and accepted by the industry that are used to assess the validity of models developed under that objective.  In practice, we see:  Development  KS / Gini used as the primary model selection tool  These evaluated on the development, hold out, and out- of-time samples  Validation  KS / ΔKS  Stability test (e.g., PSI, characteristic analysis, etc.)  Backtesting analysis 14

15 Filename 15 M odel Validation  Almost all scoring models generate KS values that reject the null that the distribution of good accounts is equal to the distribution of bads.  KS is also used to identify a specific model with the maximum separation across alternative models.  In practice, however, the difference between the maxKS and those of alternative models is never tested using statistical methods (although there are tests outlined in the literature – e.g., Krzanowski and Hand, 2011).  More importantly, once a model is selected, few modelers apply a statistical test to determine if the KS has change significantly over time to conclude the model is no longer working as expected. 15

16 Filename 16 M odel Validation  The test that have been developed, however, tend to be sensitive to sample size. Given the size of development and validation samples, very small changes may be statistically significant.  OPEN ISSUE 1: Are there tests banks can use to test for statistical significance that are not overly sensitive to sample size. 16

17 Filename 17 M odel Validation  Predictive models are developed under a model accuracy objective.  As a result, a goodness-of-fit test is required for model selection.  Common performance measures used to evaluate predictive models:  Interval Test  Chi-Square Test  Hosmer-Lemeshow (H-L) Test  Unfortunately, the goodness-of-fit tests assume defaults are independent events. If the events are dependent, the tests will reject the null too frequently. 17

18 Filename 18 M odel Validation  The Vasicek Test is an alternative test of accuracy that allows for dependence.  The Vasicek Test is designed to capture the effect of dependence on the size of the confidence bands.  Formula used to derive the confidence bands where V int is the width of the interval;  ~ N(0,1); Z.95 =1.64; and ρ – correlation. 18

19 Filename 19 Vasicek Test: An Example Vasicek Test Analysis SegmentAccountsEstimated PDActual PDVasicek Upper Bound 95% CI ρ = 0.15 ρ = 0.05ρ = 0.015 110000.000000.002000.0000030.00000 0.00005 210000.000010.000000.0000580.000040.000030.00024 310000.000080.000000.0003230.000230.000150.00062 410000.000310.001000.0012720.000870.000590.00141 510000.001020.004000.0039570.002650.001830.00299 610000.003130.008000.0114660.007600.005360.00659 710000.010030.019000.0335410.022300.016180.01620 810000.037670.063000.1078770.073920.056050.04948 910000.187980.267000.3938360.297710.245380.21220 1010000.759280.549000.9271030.864250.819190.78578

20 Filename 20 M odel Validation: Vasicek Test  If ρ is too high the bands are too wide: too many models would pass the test  ρ is not known and has to be estimated.  For point-in-time based models, ρ can be very small  For through-the-cycle based models, ρ can be large  In practice, we often see models fail the interval/Chi-square test, but pass the Vasicek test (especially when samples are large).  Open Issue 2: How do we resolve the inconsistency? 20

21 Filename 21 Sensitivity of Validation Test to Sample Size  Accuracy tests tend to reject models that  discriminate well  consistent with the expectations of the LOB  Measurement can be so precise that even a small, non-relevant difference in point estimates can be considered statistically significant. 21

22 Filename 22 Illustrative Example 22

23 Filename 23 Illustrative Example 23

24 Filename 24 Illustrative Example 24

25 Filename 25 Interval Tests with Large Samples  Conclusion:  Statistical difference: significant  Economic difference: insignificant  Solutions?  Reduce the number observations using a sample: less powerful test  Redefine the test  Interval test  Focus on capital 25

26 Filename 26 Interval Tests with Large Samples 26 (5) (4) -1%+1%0 (1) (2) (3)

27 Filename 27 Interval Test  Restate the null as an interval defined over an economically acceptable range  If the CI 1-α around the point estimate is within the in interval, conclude no economically significant difference  May want to reformulate the interval test in terms of an acceptable economic bias in the calculation of regulatory capital  Open Issue 3: How do we reconcile business and statistical significance? 27

28 Filename 28 Conclusion  Active management of model risk  Sound model development, implementation, and use of models are vital elements, and  Rigorous model validation is critical to effective model risk management.  Model Risk should be managed like other risks  Identify the source  Manage it properly 28

Download ppt "Filename Comptroller of the Currency Administrator of National Banks Managing Model Risk in Retail Scoring Dennis Glennon Credit Risk Analysis Division."

Similar presentations

Ads by Google