Presentation on theme: "Validation of Credit Risk Models"— Presentation transcript:
1Validation of Credit Risk Models March 2010Validation of Credit Risk ModelsValere BaertsBusiness Architect CPF1
2AGENDAFrom regulation to standard risk managementA generic Model Management FrameworkValidation of a Credit Risk Model: the building blocksSummary and conclusions
3AGENDAFrom regulation to standard risk managementA generic Model Management FrameworkValidation of a Credit Risk Model: the building blocksSummary and conclusions
4From regulation to standard risk management Basel one, two, three …. Так что?The Basel Accord(s) refers to the banking supervision agreements (recommendations and regulations), Basel I (1988) and Basel II (2004) issued by the Basel Committee on Banking Supervision (BCBS).The Basel Accords are implemented in ~ 100 countries worldwide. The Central Bank of Russia intends to implement also these accords but (amongst others) due to the financial crisis no exact planning is known (to me).
5From regulation to standard risk management What do these “accords” try to do?Quantitative element (called Pillar 1 in Basel II)Make sure that there is enough capital available in Bank for stress circumstances.The failure of a bank impacts the global financial system: links to other banks (Interbank market), credit crunch (macro economic effects) and “poor” depositors loosing their (part) of their savings.As a consequence, governments try to manage the risks in banks by requiring a minimum level of capital relative to its risk assets; in other words to place a limit on the ability of a bank to gear its balance sheet.Qualitative element (not in Basel I, called Pillar 2 and 3 in Basel II)Risk Governance (internal processes)and transparency (disclosure of data)
6From regulation to standard risk management How is (regulatory) capital calculated?Depending on which Basel approach the Bank chooses or has to use, there are 3 options to calculate capital:Standard approach: for smaller not complex banks- values of parameters to calculate capital are predefined by regulatorsFoundation and Advanced approach: for larger, more complex banks- values of (some) parameters to calculate capital are model based
7From regulation to standard risk management Key Drivers of credits (*) used as input into the capital calculation:the borrowers’ probabilities of default (PD)( ~ rating of borrower )(IRBF and IRBA)loss given default (LGD) and exposure at default (EaD) on an exposure-by-exposure basis (IRBA and for retail portfolio also IRBF)Maturity (IRBF and IRBA)(K) = [LGD * N [(1 - R)^-0.5 * G (PD) + (R / (1 - R))^0.5 * G (0.999)] - PD * LGD] * ( x b(PD))^ -1 . (1 + (M - 2.5) * b (PD))Risk weighted assets = 12.5 * K * EAD(*) We call this key drivers as these parameters mainly determine the credit risk related to lending products (loans, credit cards …). As such using these parameters in your credit process is “good practice” independent if the bank applies Basel II or not !!
8Borrower characteristics Facility characteristics From regulation to standard risk managementCredit RiskBorrower characteristicsFacility characteristicsWho are we lending to?Probability of DefaultHow much exposure will we have should the borrower default?Time to contractual repayment?What is the % we expect to loose taken into account recovery on collateral?MaturityExposure at DefaultLoss Given Default
9Use of Key Drivers in Credit Process From regulation to standard risk managementUse of Key Drivers in Credit Process
10How do we calculate the “Key Drivers” of Credit Risk? From regulation to standard risk managementHow do we calculate the “Key Drivers” of Credit Risk?The drivers in essence classify the customer on a scale of creditworthiness and loss probability.how likely is it that the borrower will pay back his loan? PDwhat will the exposure be when borrower defaults? EaDwhat will the loss be given the borrower defaults? LGDKey word in these questions is “will”, in other words we need to look into the future to find answers. The calculation of these drivers is equal to making a prediction.In general predictions can be “expert” based or “model” based … in both cases however the output is mainly based on historical data ( the credit experts opinion is most likely influenced with experiences from the past)…… so finally the models arrived in our story
11From regulation to standard risk management The only thing we know about the future is that it will be different. (P. Drucker)A (any) model is a simplified version of reality. It is in general a theoretical construction that represents processes by a set of variables and their relationships (in case of credit risk models the process can be described as behavior of a customer related to repayment).What can go wrong with a model ( = model risk *)Errors in the model development process (wrong data, assumptions,..)Errors in implementation of models (bugs)Wrong usage of models Due to these shortcomings the bank will refuse good customers, accept bad customers, price products to cheap, build up a bad performing portfolio, calculate to high or to low provisions etc…* Model risk can be considered as a subset of operational risk
12From regulation to standard risk management How do we manage “model risk”?Source of inspiration = Basel II regulation = good risk management practiceRegulatory Requirements on “Validation of internal estimates” (B II §§ )“Banks must have a robust system in place to validate the accuracy and consistency of rating systems, processes, and the estimation of all relevant risk components…”[Basel II, §500]“Banks must regularly compare realized default rates with estimated PD’s for each grade and be able to demonstrate that the realized default rates are within the expected range for that grade.” [Basel II, §501]“ ……So “validation” is the prescribed solution but how can we bring this concept into practice?
13AGENDAFrom regulation to standard risk managementA generic Model Management FrameworkValidation of a Credit Risk Model: the building blocksSummary and conclusions
14A generic Model Management Framework The purpose of a Model Management Framework is to:Describe a generic model life cycle (see next slide)- Allocate responsibilities to entities involved in the model life cycleWhich entity develops transactional models?Which entity develops portfolio models?Who validates the model? (internal – external)Define key functionsModel owner, Modeller, Implementation Manager, ValidatorThe function of validator is always separated from all other model management functions (also requested by regulators)For credit risk, a transactional model means any model (statistical, scorecard, expert) which aims to quantify the credit risk (measured through components as Probability of Default, Loss Given Default and Exposure at Default) of a single type of client (e.g. an SME) or product (e.g. a mortgage loan).In the context of credit risk, a portfolio model means any model which aims to quantify the risk of a portfolio of individual exposures taking into account correlations between these exposures (e.g. typically economic capital models).
16A generic Model Management Framework Model design includes:- description of model requirements- data collection and methodological choices- design of the mathematical model (including testing)definition of policies on model usage, monitoring and reportingImplementation includes both technical (ICT) and process (organizational) aspects.Once a model and its policies are implemented, they need to be monitored on an ongoing basis (minimum frequency of monitoring are part of the aforementioned policies of monitoring). The monitoring can result in minor changes (maintenance) or in a full review of the model.
17AGENDAFrom regulation to standard risk managementA generic Model Management FrameworkValidation of a Credit Risk Model: the building blocksSummary and conclusions
18Validation of Credit Risk Model: the building blocks Components of a validation exercise:Validation of the quantitative aspects- Data collection- Predictive modelRatings and calibrationValidation of the qualitative aspects- Policies- ImplementationProcess of a validation exercise
19Validation of Credit Risk Model: the building blocks Validation of the quantitative aspects – DataData are often the “Achilles' heel” in model development. For this a validation exercise should always focus in depth on potential data issues.Examples of questions to be answered:- Is a randomly selected hold-out sample used?Is the sample representative for the population targeted by the model?Is the targeted population unchanged with respect to the design?- How are good clients, bad clients defined?- How are cure events treated (observations which turn bad, then cure and become good again, and subsequently turn bad a second time .. count as only one bad observation?)- How are outcome and observation period defined?- Is the number of missing values and deleted observations acceptable?- Do all variables have an intuitive relationship with good/bad assessment?Do all variables have an acceptable predictive power?Is reject inference used?- Are the variables not highly correlated?
20Validation of Credit Risk Model: the building blocks Validation of the quantitative aspects – Predictive modelIn the next step of the validation one should independently verify the discriminatory (ranking) power of financial and non-financial sub-models and the total power on default and non-default basis.For this step the same methods are used as in scorecard/model development (ref. presentation “Credit Risk Scorecard Development”)Examples of questions to be answered:Are cut-off points of the scorecard well described and argumented?Is the power of the scorecard on the whole population acceptable?Is the test of the power based on the hold-out sample?Is the distribution of model scores acceptable?Are there significant differences between the distribution of model scores for the hold-out sample and the development sample?
21Validation of Credit Risk Model: the building blocks Validation of the quantitative aspects – Ratings and CalibrationCalibration of a model is the mapping from the scores, generated by the multifactor analysis or any other ranking technique, into PD’s.Some questions to be answered:How are default percentages calculated?Does the calibration function decrease in a monotone manner?Is the average predicted model PD for the portfolio close to the central tendency (for TTC models) or to the expected default percentage for the next year (for PIT models)?Are cut-off’s used (worst or best ratings skipped)?How are the rating classes defined?Default Rate = an observed default percentage (not to be confused with a “default frequency”, which is a natural number rather than a percentage).Central Tendency = the intrinsic long-term average one-year default percentage for a portfolio. Its value is typically estimated based on the long-term (= encompassing multiple economic cycles) average one-year default rate, possibly amended by expert input.Probability of Default = a predicted one-year default percentage for an individual customer, segment of customers or an entire portfolio. Not to be confused with the DR; there is no such thing as an “observed PD”, although the PD is always determined on the basis of DRs.Average PD predicted by a model for a sample.Monotone decreasing: it is difficult to justify that a client with a better score (i.e. for example with better financials) gets a worse PD. However, in some cases one might nevertheless observe that clients with better scores have higher DR’s than clients with lower scores.
22Validation of Credit Risk Model: the building blocks Validation of the qualitative aspects – Policies & Implementation (1)In this step of the validation exercise we mainly focus on how the developed model is brought into production. This contains as well IT aspects as business aspects (manuals, procedures etc ..).Examples of questions to be answered:Are measures taken and controls implemented to organize the access to the model in such a way that only authorized personnel have access to read, use and change the model?Are the types of transactions/counterparts/cases for which the model is intended clearly described? Are the entities that will use the model listed? Does the policy clearly mentions if and when usage of the model is mandatory?Are measures taken and controls implemented to guarantee that model inputs are correct and consistent with the data used to develop the model?
23Validation of Credit Risk Model: the building blocks Validation of the qualitative aspects – Policies & Implementation (2)…. some more questions:Does the policy describes if and under what circumstances the output of the model can be overruled?Is the model output used in an appropriate way, is specification given on how and when the output of the model should be used in the process?Is the reporting regarding model related information adequate and timely?Are the reports that will be created and distributed to monitor the model and the underlying portfolio (content, author, frequency and distribution) described?
24Validation of Credit Risk Model: the building blocks Process of a validation exercise- Read available design/review documentation- Check recommendations of previous validations, audit and regulatorsInterview the model stakeholdersIndependently recode (part of) the model- Benchmark the modeller’s analysis and calculationsPerform ad hoc testing (assess the implementation and its operational risks)- Document the validation exercise including conclusions and recommendationsInterviews with model users are means to evaluate the day-to-day use of the model. Model users are persons using the model as defined by the policy on model usage, for example in the application process, for provisioning …Benchmarking the modeller’s analysis consists of performing certain modelling steps independently from the modeller. A benchmark can consist of a complete benchmark model (including data collection, single factor analysis, multifactor analysis, calibration, etc.) or of certain modelling steps. It must be noted that a benchmark model will usually not be exactly the same as the model proposed by the modeller because modelling requires some subjective inputs. However, the results should be close.Ad hoc testing includes a large variety of tests executed to check the correct working of the model. Ad hoc tests are specific tests, often based on a sample of clients or transactions that check certain aspects related to the implementation of the policies of the model. These ad hoc tests include tests of the correct technical implementation into the IT application.An example is the manual calculation of the model results for a number of individual cases and a comparison of these results with the results obtained in the IT application. Another example is an ad hoc test of the model applicability. The model applicability can be checked, for example, by trying to assign a rating to a large corporate via a model for small businesses. Check also model access, model inputs, availablity of reports ..
25AGENDAFrom regulation to standard risk managementA generic Model Management FrameworkValidation of a Credit Risk Model: the building blocksSummary and conclusions
26Summary and conclusions Model management, translated as limiting model risk, starts with defining a clear framework containing the model life cycle, roles and responsibilities of stakeholders.While doing a validation exercise following checks are deemed mandatory:Data validation (representative, complete and appropriate)Statistical validation (discrimination, calibration and out of sample-performance)Validation of model implementation and usage (Timely, correct and correct usage?)
27Summary and conclusions Depending on the importance of the model, the depth of the validation can be extended IF data and time are available.Issues not discussed:- Timing and frequency of validation exercises.Quality of underlying processes ( how well are defaults captured? How accurate are data registered?).Reject inference.Stability of model through time (aging of portfolio and changing environment).Benchmarking against external ratings.Blind testing (experts).Stress testing.Validation techniques are and will always improve but will never take away model uncertainty. In the light of the famous Pareto principle (80/20 rule) I believe that the basics discussed today, can give bank management and regulators an acceptable level of comfort related to model risk.Depth: The depth of the validation should depend on:- The size of the portfolio covered by the model;- The perceived risk of the portfolio covered by the model;The time since the last in-depth validation took place;Timing: It is not required that the validation only starts after the design/review is finalised. It is possible to start validating the steps taken by the modeller/reviewer along the way. In this way the design/review and validation can be conducted more efficiently. The acceptability of choices made by the modeller/reviewer can be checked rather quickly which can avoid that some of the later steps need to be redone. It is noted, however, that since the different modelling steps and policies show interdependencies the final validation of the model can only be done after the design/review is finalised. An intermediate ‘nihil obstat’ of the validator to individual steps or policies is therefore only indicative and must not be interpreted as approval. As mentioned in 3.6 it is important that the meetings of the validator with persons involved in the design/review are well documented to prove the independence of the validation from the design/review.Referring to Pareto (Italian economist) who observed in 1906 that 80% of the land in Italy was owned by 20% of the population; he developed the principle by observing that 20% of the pea pods in his garden contained 80% of the peas