Validating uncertain predictions Tony O’Hagan, Leo Bastos, Jeremy Oakley, University of Sheffield

Why am I here?  I probably know less about finite elements modelling than anyone else at this meeting  But I have been working with mechanistic models of all kinds for almost 20 years  Models of climate, oil reservoirs, rainfall runoff, aero-engines, sewer systems, vegetation growth, disease progression,...  What I do know about is uncertainty  I’m a statistician  My field is Bayesian statistics  One of my principal research areas is to understand, quantify and reduce uncertainty in the predictions made by models  I bring a different perspective on model validation 6/9/20092mucm.group.shef.ac.uk

Some background  Models are often highly computer intensive  Long run times  FE models on fine grid  Oil reservoir simulator runs can take days  Things we want to do with them may require many runs  Uncertainty analysis  Exploring output uncertainty induced by uncertainty in model inputs  Calibration  Searching for parameter values to match observational data  Optimisation  Searching for input settings to optimise output  We need efficient methods requiring minimal run sets 6/9/20093mucm.group.shef.ac.uk

Emulation  We use Bayesian statistics  Based on a training sample of model runs, we estimate what the model output would be at all untried input configurations  The result is a statistical representation of the model  In the form of a stochastic process over input space  The process mean is our best estimate of what the output would be at any input configuration  Uncertainty is captured by variances and covariances  It correctly returns what we know  At any training sample point, the mean is the observed value  With zero variance 6/9/20094mucm.group.shef.ac.uk

2 code runs  Consider one input and one output  Emulator estimate interpolates data  Emulator uncertainty grows between data points mucm.group.shef.ac.uk6/9/20095

3 code runs  Adding another point changes estimate and reduces uncertainty mucm.group.shef.ac.uk6/9/20096

5 code runs  And so on mucm.group.shef.ac.uk6/9/20097

MUCM  The emulator is a fast meta-model but with a full statistical representation of uncertainty  We can build the emulator and use it for tasks such as calibration with far fewer model runs than other methods  Typically 10 or 100 times fewer  The RCUK Basic Technology grant Managing Uncertainty in Complex Models is developing this approach   See in particular the MUCM toolkit 6/9/20098mucm.group.shef.ac.uk

Validation  What does it mean to validate a simulation model?  Compare model predictions with reality  But the model is always wrong  How can something which is always wrong ever be called valid?  Conventionally, a model is said to be valid if its predictions are close enough to reality  How close is close enough?  Depends on purpose  Conventional approaches to validation confuse the absolute (valid) with the relative (fit for this purpose)  Let’s look at an analogous validation problem 6/9/20099mucm.group.shef.ac.uk

Validating an emulator 6/9/2009mucm.group.shef.ac.uk10  What does it mean to validate an emulator?  Compare the emulator’s predictions with the reality of model output  Make a validation sample of runs at new input configurations  The emulator mean is the best prediction and is always wrong  But the emulator predicts uncertainty around that mean  The emulator is valid if its expressions of uncertainty are correct  Actual outputs should fall in 95% intervals 95% of the time  No less and no more than 95% of the time  Standardised residuals should have zero mean and unit variance  See Bastos and O’Hagan preprint on MUCM website

Validation diagnostics 6/9/2009mucm.group.shef.ac.uk11

Validating the model 6/9/2009mucm.group.shef.ac.uk12  Let’s accept that there is uncertainty around model predictions  We need to be able to make statistical predictions  Then if we compare with observations we can see whether reality falls within the prediction bounds correctly  The difference between model output and reality is called model discrepancy  It’s also a function of the inputs  Like the model output, it’s typically a smooth function  Like the model output, we can emulate this function  We can validate this

Model discrepancy 6/9/2009mucm.group.shef.ac.uk13  Model discrepancy was first introduced within the MUCM framework in the context of model calibration  Ignoring discrepancy leads to over-fitting and over-confidence in the calibrated parameters  Understanding that it is a smooth error term rather than just noise is also crucial  To learn about discrepancy we need a training sample of observations of the real process  Then we can validate our emulation of reality using further observations  This is one ongoing strand of the MUCM project

Beyond validation 6/9/2009mucm.group.shef.ac.uk14  An emulator (of a model or of reality) can be valid and yet useless in practice  Given a sample of real-process observations, we can predict the output at any input to be the sample mean plus or minus two sample standard deviations  This will validate OK  Assuming the sample is representative  But it ignores the model and makes poor use of the sample!  Two valid emulators can be compared on the basis of the variance of their predictions  And declared fit for purpose if the variance is small enough

In conclusion 6/9/2009mucm.group.shef.ac.uk15  I think it is useful to separate the absolute property of validity from the relative property of fitness for purpose  Model predictions alone are useless without some idea of how accurate they are  Quantifying uncertainty in the predictions by building an emulator allows us to talk about validity  Only valid statistical predictions of reality should be accepted  Model predictions with a false measure of their accuracy are also useless!  We can choose between valid predictions on the basis of how accurate they are  And ask if they are sufficiently accurate for purpose

Advertisement 6/9/2009mucm.group.shef.ac.uk16  Workshop on emulators and MUCM methods  “Uncertainty in Simulation Models”  Friday 10th July 2009  10.30am - 4pm  National Oceanography Centre Southampton   Please register with Katherine Jeays-Ward  by 3rd July 2009  Registration is free, and lunch/refreshments will be provided