William Cook Abusing statistics in retail banks, and its contribution to the banking crisis
Introduction Provide modelling support to banks and building societies within retail credit risk departments Have worked with likes of Nationwide, Lloyds Banking Group... Bradford & Bingley, Northern Rock Based on standard UK industry practices, will present here two examples of the abuse of statistics that contributed to the banking crisis 2
Background to credit risk Within banks there are two buffers against losses: Provisions – against short term forecast losses Capital – a reserve for the worst case scenario Provisions can be difficult to forecast, capital can be even harder to estimate Miscalculation of retail capital contributed to the downfall of various banks in the UK 3
Capital calculations Regulation for capital calculation comes from the FSA Most large banks have opted for the “Internal Ratings Based” approach This allows banks to make their own estimates, rather than using standard benchmark ratios Since inception, the FSA has been keen for capital to be calculated at an individual account level under IRB A key component of these account level models is the Probability of Default, or PD Important for capital calculations that the PD can be ‘accurately’ modelled 4
Credit risk through time Percent of UK mortgages 6-12 months in arrears 5
Problem 1 – The Gini coefficient Account level PD modelling performed with logistic regression Sample consists of all those who are not in default Outcome is 0 for those that do not default in the following year and 1 for those that do Independent variables come from many sources and include information such as age, performance on other products and credit reference agency data Model power typically measured with the Gini 6
Gini value = 0.72 Gini – example curve 7
Gini – overlapping distributions Comparing ‘good’ and ‘bad’ distributions 8 badsgoods
Gini - two way table example Can also look at 2 way table of predicted versus actual If we wish to match total predicted with total actuals, need to find cut-off in predicted values Example with 10,000 customers, 5,000 are ‘bad’ Gini = 0.72, sensitivity = Actual GoodBadTotal PredictedGood3,8911,1084,999 Bad1,1093,8925,001 Total5,000 10,000
Gini – two way table example 2 In reality, typically trying to model an overall bad rate at 5% or less Another example with 10,000 customers, but only 500 bad Keep Gini = 0.72 Sensitivity now depends on prediction cut-off: If same cut-off used, remains same (as well as specificity) If new optimal cut-off used, sensitivity drops 10
Gini – two way table example 2 Same cut-off, sensitivity = 0.77 New optimal cut-off, sensitivity = Actual GoodBadTotal PredictedGood7, ,485 Bad2, ,515 Total9, ,000 Actual GoodBadTotal PredictedGood9, ,500 Bad Total9, ,000
Gini conclusions Ginis reported for credit risk models are often flattering compared to two way tables Fundamentally trying to predict an outcome that is difficult to forecast Often default is driven by future unemployment, which may not be picked up by independent variables 12
Problem 2 – Cross-sectional and time-series relationships Calculating the correct amount of capital is very much a time-series problem Logistic regression models used for predicting default are cross-sectional Well known in statistics that relationships that hold for one type of analysis do not necessarily hold for the other Simple example with earnings and savings 13
Cross-sectional and time-series example Income and savings time series 14
Macro driver of credit risk Comparing unemployment and mortgage arrears 15
Conclusions Whilst logistic regression modelling does have its place in credit risk, the apparent power of such models did deceive the regulator and the industry Since March of this year, the FSA has been making big changes in the way in which capital should be calculated for IRB banks Rightly, there have been moves away from account level methodology to focus on long run time-series performance data 16