1. THE ADVANCED IRB FORUM Monday 23 June Kevin Ryan

2. AREAS TO BE COVERED Issues in validating Internal Ratings systems How are supervisors reacting – emerging thoughts The FSA Consultative Paper –Sections on Validation, External Models & Data, Data Quality, Assessment Horizon & PD estimation

3. NOT TO BE COVERED Detailed ‘how to do it’ to get your IRB systems approved by the FSA We do not know enough at this stage –“We recognise a lot of work remains in this complex area with many challenges for firms and us” Responsibility of firms to validate their rating systems –Firms ‘validate’; supervisors ‘approve’ (or certify)

4. STYLISED SUPERVISORS’ VIEW OF RATING SYSTEMS Data and validation a challenge! Good work has been done, but not been part of the mainstream activity of the firms, so –Even in the best firms, few people understand the issues –Knowledge and oversight by senior management and control functions quite low –Resources for improvements can be difficult to get, so trade- offs made –Limited appetite for fresh thinking or challenging assumptions Appetite for short cut solutions, eg agency ratings

5. A MODEL IS PROVED BY ITS PERFORMANCE ? SMALL SAMPLE SIZE In the common binomial test, construct a confidence interval around estimated PD Suppose a PD estimate is 100bp, and you want to be 95% confident that actual PD is between 80 and 120bp

6. A MODEL IS PROVED BY ITS PERFORMANCE ? SMALL SAMPLE SIZE In the common binomial test, construct a confidence interval around estimated PD Suppose a PD estimate is 100bp, and you want to be 95% confident that actual PD is between 80 and 120bp You need 9500 borrowers in that grade

7. A MODEL IS PROVED BY ITS PERFORMANCE ? SMALL SAMPLE SIZE Using the common binomial test, what levels of true PD are consistent with zero observed defaults, at a 95% confidence level? If you have 500 borrowers in a grade, any PD up to 1.25% If you have 80 in a grade, a PD as high as 6% or 7% (Binomial test doesn’t work if PD x Borrowers below 5)

8. A MODEL IS PROVED BY ITS PERFORMANCE ? DEFAULTS ARE CORRELATED The distribution you can expect from defaults is many periods with actual defaults below the mean, and fewer periods (typically clustered together) with actual defaults well above the mean IMPLICATIONS Confidence bounds wider than binomial More difficult to interpret

9. A MODEL IS PROVED BY ITS PERFORMANCE ? IMPLICATIONS “Regardless of the efforts used by banks, …there will still remain some portfolios where there will likely never be sufficient default data to calibrate PDs with any degree of statistical significance” Internal Ratings Validation Study “We’re going to need to be pragmatic for some years to come”

10. WHAT CAN BE DONE? BROADER APPROACH TO VALIDATION, TO SUPPLEMENT OUTCOMES ANALYSIS Logic and conceptual soundness of the approach Statistical testing prior to use Monitoring of process – are the methods being applied as intended Benchmarking – compare to relevant alternatives David Wright, FRB, 19 June

11. WHAT CAN BE DONE? MORE ON OUTCOMES ANALYSIS, INCLUDING More exploration and use of tolerance levels by firms, which requires Better understanding of distribution of expected defaults Supervisory interest in information content of transition matrices What can be learnt from other industries? –Insurance experience of modeling rare events?

12. FSA APPROACH – FSA CP KEY POINTS Accountability –Firms responsibility to validate & submit documentation with IRB application –Application to be signed by chief executive We expect he will have similar questions to supervisors Independence –Approved by senior committee –Independent staff to participate

13. FSA APPROACH – FSA CP KEY POINTS Scope –To cover all portfolios, but depth will vary with significance Materiality considerations –Methods may vary, especially by portfolio If more reliance can be placed on back-testing we should need less additional evidence If less reliance on back-testing, more additional evidence is needed

14. FSA APPROACH – FSA CP KEY POINTS Scope –Must assess the accuracy of the overall output of the system Not just the inputs Will need to take account of overrides and judgmental adjustments to any underlying statistical models –Based on statistical analysis of rating system, related internal data, and third party and publicly available information

15. FSA APPROACH – FSA CP Coverage –Full documentation –Clarity on what rating system aiming to predict, and expected distributions –Take full account of adjustments between unbiased estimates and those used in regulatory capital calculation Conservatism Cyclical effects Double default effects

16. FSA APPROACH – FSA CP Coverage –Clearly set out standards of control –Clearly set out limitations of approach Assume that there will be candour –Include work to demonstrate both ability to rank into grades (discriminate) and estimate a PD etc (calibrate) –Include steps to be taken in event quantitative tests for power and accuracy are breached

17. FSA APPROACH – FSA CP Discrimination/ranking –Firm must justify that system shows a ‘high degree of power in line with industry norm for portfolios of that nature’ –We do not specify what test should be used, eg Gini or other, or its level But we consult on such a test –Expect firms to strive for best model they can, other things being equal

18. FSA APPROACH – FSA CP Discrimination/ranking - issues –Some see discrimination as key to model building –Some research that high discrimination needed to achieve accurate PD measures –But levels dependent on features of portfolio, number of defaults –High is good, but may be over-fitting –Firms must use targets, but reluctant to admit to them –Importance of expertise to interpret

19. FSA APPROACH – FSA CP Calibration –Standardised ‘scorecard’ proposed –Firm to justify its estimate against own historic experience and external sources –Take account of factors which may expect to lead to differences; eg conservatism, cycle effects –We do not specify tests for assessing accuracy But we consult on such a test –If actuals not consistent with estimates, firms must justify differences and/or take steps to improve

20. FSA APPROACH – FSA CP Data quality –“Data quality was not seen as the major obstacle to validation in the long term” Internal Ratings Validation Study –Some evidence that cleaning the data produces more powerful models, while greater accuracy self evident –Missing defaults? –“We recognise that the data accuracy challenge for IRB is significant” FSA CP –“We propose quantifiable targets to cover completeness and accuracy that will rise over time”

21. FSA APPROACH – FSA CP External models and data –In principle supportive as can supplement internal data and models; also external vendors may operate to higher standards than firms –But external models carry explicit risk of black box, with limitations not known and application to inappropriate portfolios –Limit to how much vendors will reveal –We are trialing ‘vendor grid’ which would standardise information to be provided – aimed at reducing risks

22. FSA APPROACH – FSA CP Pooled data –In principle supportive as, like external data, can supplement internal experience, and without commercial confidentiality issues of external vendors –“Issues and challenges facing pooling initiatives Consistent data definitions for all Legal restrictions – data protection and confidentiality Potential increase in systemic risk” Internal Ratings Validation Study

23. FSA APPROACH – FSA CP Expert judgment –“There is a feeling that not enough time has been spent on discussing acceptable validation techniques for these types of systems” Internal Ratings Validation Study –Will look to accommodate, but validation challenges increase –Are there some cases where it is not feasible to produce quantified estimates? –Or use ranking with conservative estimates of losses

24. BENCHMARKING A complement to back-testing Conceptually two types, although differences between them may be blurred in practice –Relative benchmarks – compare estimates between firms to identify outliers –Absolute benchmarks - compare with an external benchmark given some credibility We propose some element of benchmarking in our proposed approaches to discrimination (industry standard) and calibration (comparison with external sources)

25. BENCHMARKING POSSIBLE INITIATIVES Private sector services to benchmark ratings and/or estimates Some international interest in requiring firms to rate ‘test portfolios’ FSA considering targeted benchmarking at obligor level where back-testing difficult and amounts large –Very early stage of thinking but would welcome feedback –Could be run by industry, FSA or other body

26. SOME VALIDATION CHALLENGES Scale of task given possible number of models –Materiality to firm v materiality to the market –Importance of consistency Need for firms and supervisors to increase expertise –Even statistical models require expert judgment –How much do supervisors need to know? Can we give enough guidance to allow objective self-assessment? What is the scope and appetite for improved standards?

27. SOME VALIDATION CHALLENGES What is the supervisory standard? –Can we identify a definite pass or a definite fail? Benchmarks which must be beaten to qualify, or if beaten are sufficient Implications for expert judgment approaches, or methods which ‘perform’ less well? Other objectives may be avoidance of systemic risk, and allowing entry to stimulate competition –How do we incentivise firms to improve? Rising standards or Pillar 2 requirements –Too much conservatism will take away incentives LGD and EAD?