Presentation is loading. Please wait.

Presentation is loading. Please wait.

Building an Emulator. EGU short course – session 22 Outline Recipe for building an emulator – MUCM toolkit Screening – which simulator inputs matter Design.

Similar presentations

Presentation on theme: "Building an Emulator. EGU short course – session 22 Outline Recipe for building an emulator – MUCM toolkit Screening – which simulator inputs matter Design."— Presentation transcript:

1 Building an Emulator

2 EGU short course – session 22 Outline Recipe for building an emulator – MUCM toolkit Screening – which simulator inputs matter Design – where to run the simulator Model structure – mean and covariance functions Estimation / inference – building the emulator Validation – making sure the emulator is OK Possible extensions multiple outputs dynamic simulators Bayes linear methods Summary of emulation

3 The ‘standard’ problem The MUCM toolkit recipe

4 Step 0: Know your simulator Before attempting to create an emulator it is important you understand your simulator what are the plausible input ranges what constraints are there in input combinations what is the output behaviour like Ideally you may wish to elicit beliefs about the distributions of the inputs if these are not known at least ranges are needed for all inputs

5 EGU short course – session 25 Step 1: Screening – active inputs All serious simulators require more than one input the norm is anything from a few to thousands all of the basic emulation theory in the toolkit assumes multiple inputs Large numbers of inputs pose computational problems dimension reduction techniques have been developed output typically depends principally on a few inputs Screening seeks to identify the most important inputs for a given output most often the Morris method is used, which is a cheap sensitivity analysis approximation

6 EGU short course – session 26 Screening: the Morris method Basic idea: develop a design that changes one input at a time, while filling space Morris designs based on a series of repeated trajectories which change only one input each step Compute: average of the elementary effects (μ) variance of the elementary effects (σ) no effectlinear effect non-linear effect

7 Step 2: Design Design is all about choosing where to run the simulator to learn a good emulator we also need to consider validation and calibration There are many options for design and many issues in the absence of additional information space-filling designs are used grids are infeasible for all but trivial simulators

8 EGU short course – session 28 Training sample design To build an emulator, we use a set of simulator runs our training data are y 1 = f(x 1 ),..., y n = f(x n ) x 1, x 2,..., x n are n different points in the space of inputs this set of n points is a design A good design will provide us with maximum information about the simulator and hence an emulator that is as good as possible

9 EGU short course – session 29 Latin hypercube designs LHC designs use n values for each input combining randomly Advantages doesn’t necessarily require a large number of points nothing lost if some inputs are inactive Disadvantages random choice may not produce an even spread of points need to generate many LHC designs and pick the best

10 EGU short course – session 210

11 EGU short course – session 211 Some more design choices Various formulae and algorithms exist to generate space- filling designs for any number of inputs the Sobol sequence is often used quick and convenient not always good when some inputs are inactive Optimal designs maximise/minimise some criterion e.g. maximum entropy designs can be hard to compute, often not massive gains Hybrid designs try to satisfy two criteria space-filling but also having a few points closer together in order to estimate correlation lengths well

12 EGU short course – session 212 Step 3: Building the emulator In deciding on the structure of the emulator we have many choices to make: the mean function the covariance function the prior specifications There are no universal solutions here, so judgement and validation play an important role

13 EGU short course – session 213 The technical part (overview!) The emulator is a Gaussian process with: mean function m(x) = h(x) T β, with h(x) typically [1,x] covariance function σ 2 c(x,x’) = σ 2 exp-{(x-x’) T C(x-x’)} C is a diagonal matrix of inverse length scales 1/ δ  thus the conditional distribution of the simulator output, y, given the input, x and parameters (β, σ 2, δ) is multivariate normal The choices we make can be important

14 EGU short course – session 214 The GP mean function We can use this to say what kind of shape we would expect the output to take as a function of the inputs Most simulator outputs exhibit some overall trend in response to varying a single input so we usually specify a linear mean function slopes (positive or negative) are estimated from the training data the emulator mean smoothes the residuals after fitting the linear terms We can generalise to other kinds of mean function if we have a clear idea of how the simulator will behave the better the mean function the less the GP has to do

15 EGU short course – session 215 Example Simulator is solid line Dashed line is linear fit Blue arrows indicate fitted residuals Without the linear mean function, we’d have a horizontal (constant) fit and larger residuals leading to larger emulator uncertainty

16 EGU short course – session 216

17 The GP covariance function The covariance function determines how ‘wiggly’ the response is to each input There’s a lot of flexibility here, but standard covariance functions have a parameter for each input these ‘correlation length’ parameters are also estimated from the training data but some care is needed For predicting output at untried x, correlation lengths are important they determine how much information comes from nearby training points and hence the emulator accuracy

18 Prior distributions Prior information enters through the form of the mean function and to a lesser extent the covariance function But we can also supply prior information through the prior distributions for slope/regression parameters and correlation lengths also the overall variance parameter Putting in genuine prior information here generally improves emulator performance compared with standard ‘non-informative’ priors e.g.

19 EGU short course – session 219 Step 4: Learning the emulator We normally proceed using Bayesian inference just how Bayesian depends on size of problem ideally we would ‘integrate out’ all unknown parameters, but this can be difficult, requiring MCMC Details are on the toolkit, but in summary typically one can integrate out the regression coefficients (β) and variance parameters(σ 2 ) optimise (maximum likelihood, or MAP) the covariance length scales (δ) Ignoring uncertainty in length scales can be a problem if they are not well identified, but typically the mean function does most of the work

20 EGU short course – session 220

21 EGU short course – session 221 Prediction with the emulator Once the (hyper)-parameters of the emulator have been learnt (or integrated out) one can use the emulator to predict at a new input what the simulator output would have been this is always a predictive distribution

22 EGU short course – session 222

23 EGU short course – session 223 Step 5: Validating the emulator Validating the emulator is essential full probabilistic assessment of fitness for purpose first examine the standardised residuals, with +/- 2 std intervals visual assessment is often very helpful and provides diagnostic information

24 EGU short course – session 224 What is validation? What does it mean to validate an emulator? compare the emulator’s predictions with the simulator output make a validation sample of runs at new input configurations the emulator mean is the best prediction and is always wrong but the emulator predicts uncertainty around that mean The emulator is valid if its expressions of uncertainty are correct actual outputs should fall in 95% intervals 95% of the time no less and no more than 95% of the time standardised residuals should have zero mean and unit variance See Bastos and O’Hagan preprint on MUCM website

25 Measures for validation The Mahalanobis distance on a test set accounts for the predictive covariance on the test set follows an F-distribution so we can check the value is close to the theoretical one for a given test set size A useful diagnostic is the pivoted Cholesky decomposition of the predictive covariance Suggests non-stationary / poor predictive varianceSuggests poor length scale / covariance function Should follow a t distribution

26 Extensions What types of simulator are amenable to emulation?

27 EGU short course – session 227 Many outputs Most simulators also produce multiple outputs for instance, a climate simulator may predict temperature on a grid, etc. Usually, for any given use of the simulator we are interested in just one output so we can just emulate that one, particularly if it is some combination of the others, e.g. mean global surface temperature But some problems require multi-output emulation again, there are dimension reduction techniques All described in the MUCM toolkit

28 EGU short course – session 228 Multi-output emulators When we need to emulate several simulator outputs, there are a number of available approaches single output GP with added input(s) indexing the outputs for temperature outputs on a grid, make grid coordinates 2 additional inputs independent GPs multivariate GP independent GPs for a linear transformation e.g. principal components possibility for dimension reduction These are all documented in the MUCM toolkit

29 EGU short course – session 229

30 EGU short course – session 230 Dynamic emulation Many simulators predict a process evolving in time at each time-step the simulator updates the system state often driven by external forcing variables at each time-step climate models are usually dynamic in this sense We are interested in emulating the simulator’s time series of outputs the various forms of multi-output emulation can be used or a dynamic emulator, emulating the single time-step and then iterating the emulator Also documented in the MUCM toolkit

31 EGU short course – session 231

32 EGU short course – session 232 Stochastic emulation Other simulators produce non-deterministic outputs running a stochastic simulator twice with the same input x produces randomly different outputs Different emulation strategies arise depending on what aspect of the output is of interest interest focuses on the mean output has added noise which we allow for when building the emulator interest focuses on risk of exceeding a threshold emulate the distribution and derive the risk emulate the risk This is not yet covered in the MUCM toolkit

33 EGU short course – session 233 Bayes linear methods So far assumed a fully Bayesian framework But there is an alternative framework – Bayes linear methods based only on first and second order moments means, variances, covariances avoids making assumptions about distributions its predictions are also first and second order moments means, variances, covariances but no distributions The toolkit contains theory and procedures for Bayes linear emulators

34 EGU short course – session 234

35 EGU short course – session 235 Bayes linear emulators Much of the mathematics is very similar a Bayes linear emulator is not a GP but gives the same mean and variance predictions for given correlation lengths, mean function parameters although these are handled differently but the emulator predictions no longer have distributions Compared with GP emulators advantages – simpler and may be feasible for more complex problems disadvantages – absence of distributions limits many of the uses of emulators compromises made

36 Summary and Limitations Why emulation is not always a silver bullet

37 EGU short course – session 237 Some caveats on emulation Not all simulators are suitable for emulation with very large numbers of (>50) outputs need specific emulators and large training sets for the problem you are solving are all outputs needed? for dynamic simulators with high dimensional state spaces there remain computational issues with discrete inputs and outputs Gaussian processes are not well suited But these issues are being addressed actively in research projects across the world including MUCM

38 EGU short course – session 238 Typical sequence of emulation Define the problem you want to solve, identify the simulator Identify the inputs, define ranges and screen to select Design the training set and run the simulator Choose the emulator (mean and covariance) and define priors Train the emulator using the training set and inference method Use the emulator and if necessary refine Validate the emulator and if necessary refine Modify the simulator or refine it, maybe using observations

39 EGU short course – session 239 Summary Before you emulate know your simulator! Think carefully about the problem you really want to solve emulation is a tool to solve interesting problems and not an aim in itself The more prior knowledge you bring the easier the task will be choosing mean and covariance, eliciting priors Spend time on validation and refinement Building an emulator will help you understand your simulators … not replace them!

Download ppt "Building an Emulator. EGU short course – session 22 Outline Recipe for building an emulator – MUCM toolkit Screening – which simulator inputs matter Design."

Similar presentations

Ads by Google