1. Quantifying Model Specification Risk in Practice
Actuarial Teachers’ and Researchers’ Conference
Edinburgh, 2nd December 2014
Alan Forrest
RBS Group Risk Analytics Independent Model Validation
Information Classification –

2. Disclaimer Disclaimer
The opinions expressed in this document are solely the author’s and do not necessarily reflect those of The Royal Bank of Scotland Group or any of its subsidiaries.
Any graphs or tables shown are based on mock data and are for illustrative purposes only.

3. Overview
Background
Background to Model Risk – the risk of using a model.
Model Risk Quantification
Model specification risk as a Data-shift Problem.
Quick estimates of specification risk using geometric and information-theoretic approaches to the data-shift problem.

4. Model Risk Background The US Regulator (Fed / OCC 2011-12a )
“The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.”
Using a model presents a risk.
FSA - Turner Review - March 2009
“Misplaced reliance on sophisticated maths”
The assumptions and limitations of the models were not communicated adequately to the pricing and lending decision-makers.
BoE - The Dog and the Frisbee – Haldane, August 2012
“… opacity and complexity… It is close to impossible to tell whether results from [internal risk models] are prudent.”
If we cannot say why we trust a model, are we right to use it?

5. Model Risk Background Fed / OCC 2011-12a
“Model Risk should be managed like other types of risk.”
Identify
Quantify / assess
Act / manage
Monitor
Focus on specification risk:
The part of model risk connected with model selection.
Model risk also includes risks of model implementation, use and interpretation.
Quantification of specification risk:
How differently could the model have been built under different conditions?

6. Example Model Risk
A Probability of Default model is proposed for implementation
The model includes a factor W that has 20% missing values.
The missing values have been filled in, all with the same “mean” value, and the preferred model has been built with this imputed factor.
Missing values tend to be associated with older accounts.

7. Example Model Risk
Hypothetical alternative model builds – based on different development data:
With “missing” as a special class.
With a different method of imputation
Explore the possible bias over time by forcing (at random) new missing values among more recent accounts.

8. Quantifying Model Risk – Bottom Up
The key to specification risk quantification is sensitivity analysis;
and most sensitivity analysis can be expressed as a Data-shift Problem.
If the data used to build a model shifts, how far and in what way does the model shift?
In practice, we could have 100 data-shifts to test, and models require much resource to build.
Modellers need a quick and reliable way of assessing the likely impact of data shifts without needing to build models:
Prioritising analysis;
Getting immediate assurance on shifts that are immaterial;
Making the appropriate model changes.

9. Geometry and Model Sensitivity
The sensitivity of a model to data-shift has a geometric interpretation.
Model fitting looks “geometric”: the model is found “closest” to the data.
Geometry and curvature expresses data-shift sensitivity.
Original data
Model space
Just right
Shifted data
Type 3 (or type 0) Error
Over-fitting
Over-sensitive / discontinuous

10. Simplifications and Conventions
Banking Book Credit Risk Models allow the following assumptions:
All factors and outcomes are categorical / classed / discrete
The development data-set is completely described by its frequency table: i.e. whole number entries in a finite contingency table.
The space of all data is finite dimensional.
Fixed-effects regression based on an exponential family of distributions
Includes all the classical regressions: logistic, multinomial, poisson etc.
Maximum Likelihood Estimation (MLE)
The culture and implementation of Banking Book Credit Risk Models allow the following assumptions
All factors and outcomes are categorical / classed / discrete
The development data-set is completely described by its frequency table: i.e. whole number entries in a finite contingency table.
The space of all data is finite dimensional
Fixed-effects regression on exponential family distributions
Maximum Likelihood Estimation (MLE)
A model is a description of the data on which it is developed:
a model is another point in data space.
This point of view makes no distinction (yet) between input and output variables.
The model / description is chosen from a limited set of possible descriptions – the model space
This model space is a subspace of data space, chosen for its convenience, simplicity or usefulness
What about inputs and outputs?
Isn’t a model an algorithm? Why introduce a “description” that comes with an input distribution as well as an algorithm to derive an output.
But, if the model space covers all input populations, then this allows* a model outcome to be defined for any input population.
* non-trivial theorem

11. Simplifications and Conventions
A model is a description of the data on which it is developed:
a model is another point in data space.
The preferred model is chosen from a limited set of possible descriptions – the model space.
This model space is a subspace of data space, chosen for its convenience, simplicity or usefulness.
What about inputs and outputs?
Some dimensions of the data-space are inputs, others are outputs.
If the model space covers marginally all input populations, then* the development data and the MLE optimised description have the same input population distributions: this allows a model outcome to be defined for any input population.
* non-trivial theorem
The culture and implementation of Banking Book Credit Risk Models allow the following assumptions
All factors and outcomes are categorical / classed / discrete
The development data-set is completely described by its frequency table: i.e. whole number entries in a finite contingency table.
The space of all data is finite dimensional
Fixed-effects regression on exponential family distributions
Maximum Likelihood Estimation (MLE)
A model is a description of the data on which it is developed:
a model is another point in data space.
This point of view makes no distinction (yet) between input and output variables.
The model / description is chosen from a limited set of possible descriptions – the model space
This model space is a subspace of data space, chosen for its convenience, simplicity or usefulness
What about inputs and outputs?
Isn’t a model an algorithm? Why introduce a “description” that comes with an input distribution as well as an algorithm to derive an output.
But, if the model space covers all input populations, then this allows* a model outcome to be defined for any input population.
* non-trivial theorem

12. Geometry and Model Fitting
Fitting a model to data: a log-linear example mw = ceaw
MLE fit
Data

13. Information Principles
Dual optimisation principles for exponential families
The red model spaces are generated multiplicatively / ‘tropically’.
Data points are drawn to the MLE model along the blue spaces, which are linear.
Principle of Maximum Likelihood
The model that maximises likelihood, for data d, is the point, m, in red space that minimises KL divergence I(d,m).
Principle of Maximum Entropy
If m’ is any model in the red space then, within each blue space, the red point m minimises KL divergence, I(m,m’) .
Principles of Inference
if m is the MLE fit to data d, and m’ is another model from the model space, then I(d,m’) = I(d,m) + I(m,m’) .

14. Geometry and Model Sensitivity
Geometry of the data space
The natural metric is ds2 = Sw dxw2 / xw
Locally equal to Kullback-Leibler divergence and to Hellinger distance.
“Local Chi-squared”.
“Boot-strapping geometry”.
Isometric to a portion of a sphere.
2x = u2 connects this space isometrically with Euclidean space.
The model fitting foliations are orthogonal in this metric.
The model space curvature reflects true model sensitivity.
The data shift problem is geometric and mathematically rich
Centsov (1965, et seq.), Efron (1978 et seq.), Amari et al. (1982 et seq.), Lauritzen (1980s), Critchley et al. (1993 et seq.), etc.
Recent developments in Machine Learning, by Kanamori, Shimodaira (2009) and others, are particularly relevant to sensitivity analysis.
Hellinger distance s2(x,x’) = Sw (SQRT(xw) - SQRT(x’w))2 where xw , x’w are cell frequencies.
This metric is naturally connected with
Kullback-Leibler -divergence : ds2 = KL(x,x+dx) = KL(x+dx,x) up to second order;
Chi-squared : ds2 = Sw dxw2 / xw ;
Spherical geometry : 4x = u2 ;
Bootstrapping variation.
In this metric, the model space curvature is true and reflects model specification sensitivity accurately.

15. Efficient Sensitivity Analysis
A sensitivity principle implied by boot-strapping
For large development samples, the standard error ellipsoid is sufficient to describe model sensitivity to data shifts.
Model Standard Error Ellipsoid = image of data ds-ball
Data Bootstrapping Ellipsoid = ds-ball
Scale = Chi-sq (df = dimension) / (2N)
Model Prediction Error Ellipsoid = ds-ball

16. Geometry and Model Sensitivity
Example: Sensitivities for factor with 20% missing values
Distances (squared) between hypothetical alternative datasets, computed in spherical metric from marginals illustrated.
Additional distance estimated by KL information value relative to marginals.
0.2536
0.0443

17. Managing Sensitivities
Model Risk - 20% missing values - example revisited :
PD has been built from a pool of 12 classed factors:
Dimension of the data space (roughly = number of cross-tab cells), D = 50,000.
PD model built by MLE on sample of N = 500,000 records.
Bootstrap scale is D/2N =
Sensitivity Test
Data Shift
distance-squared
Observation
Action
Force missing values among new accounts
0.008
Correlation of missingness with age of accounts is unlikely to cause significant change in model build.
No need to investigate.
Missing at Random
v. Mean Allocation
0.045
These two imputation options are likely to result in same factor selections, but different factor weights.
Low priority
Missing as a separate category
0.254
Model build is likely to be materially different if missing is treated as a separate category.
High priority

18. Conclusions Quantifying Model Risk
Quantitative model risk assessment is needed for
consistent management and maintenance of a bank’s models, and
effective communication of model weaknesses and limitations to decision-makers and users.
The key to bottom up quantitative model risk assessment is sensitivity analysis.
The key to practical sensitivity analysis is data-shift.
Data-shift is a deep problem, well-known in geometric statistics, information theory and artificial intelligence, and rich in mathematical interest.
Simplified approximate solutions to the data-shift problem can be used in practice to quantify and prioritise model risk assessment in banking book credit risk.

19. An incomplete bibliography
The geometric approach to statistics is rich and well-established, classically starting with Rao in the 1940s
Its connection with information and KL divergence is also developed in great generality
Centsov (1965, et seq.), making geometric the original developments of Kullback and Leibler.
Efron (1978 et seq.), Lauritzen (1980s), Critchley et al. (1993 et seq.), etc.
Amari et al. (1982 et seq.) develops Akiake’s insights using differential geometry
The application to the data shift problem is comparatively recent but growing
Recent developments in Machine Learning, by Kanamori, Shimodaira (2009) and others, are particularly relevant to sensitivity analysis.
Hulse et al. (2013) recently explored its implications in Financial Modelling.