1. Uncertainty Analysis Using GEM-SA

2. GEM-SA course - session 42 Outline Setting up the project Running a simple analysis Exercise More complex analyses

3. Setting up the project

4. GEM-SA course - session 44 Create a new project Select Project -> New, or click toolbar icon Project dialog appears We’ll specify the data files first

5. GEM-SA course - session 45 Files The “Inputs” file contains one column for each parameter and one row for each model training run (the design) The “Outputs” file contains the outputs from those runs (one column, in this example) Using “Browse” buttons, select input and output files

6. GEM-SA course - session 46 Our example We’ll use the example “model1” in the GEM-SA DEMO DATA directory This example is based on a vegetation model with 7 inputs RESAEREO, DEFLECT, FACTOR, MO, COVER, TREEHT, LAI The model has 16 outputs, but for the present we will consider output 4 June monthly GPP

7. GEM-SA course - session 47 Number of inputs Click on Options tab Select number of inputs using Or click “From Inputs File”

8. GEM-SA course - session 48 Define input names Click on “Names …” The “Input parameter names” dialog opens Enter parameter names Click “OK”

9. GEM-SA course - session 49 Complete the project We will leave all other settings at their default values for now Click “OK” The Input Parameter Ranges window appears

10. GEM-SA course - session 410 Close and save project Click “Defaults from input ranges” button Click “OK” Select Project -> Save Or click toolbar icon Choose a name and click “Save”

11. Running a simple analysis

12. GEM-SA course - session 412 Build the emulator Click to build the emulator A lot of things now start to happen! The log window at the bottom starts to record various bits of information A little window appears showing progress of minimisation of the roughness parameter estimation criterion The “Main Effects” tab is selected, in which several graphs are drawn Progress bar at the bottom

13. GEM-SA course - session 413 Focus on the log window The “Main Effects” and “Sensitivity Analysis” tabs are concerned with SA, and will be considered in the next session We are interested just now simply in Uncertainty Analysis (UA) The “Output Summary” tab contains all we need and more But the key things can be seen more simply in the log window at the bottom Diagnostics of the emulator build The basic uncertainty analysis results

14. GEM-SA course - session 414 Emulation diagnostics Note where the log window reports … The first line says roughness parameters have been estimated by the simplest method The values of these indicate how non-linear the effect of each input parameter is Note the high value for input 4 (MO) Estimating emulator parameters by maximising probability distribution... maximised posterior for emulator parameters: precision = sigma- squared = 0.342826, roughness = 0.217456 0.0699709 0.191557 16.9933 0.599439 0.459675 1.01559

15. GEM-SA course - session 415 Uncertainty analysis – mean Below this, the log reports So the best estimate of the output (June GPP) is 24.1 (mol C/m 2 ) This is averaged over the uncertainty in the 7 inputs Better than just fixing inputs at best estimates There is an emulation standard error of 0.062 in this figure Estimate of mean output is 24.145, with variance 0.00388252

16. GEM-SA course - session 416 Uncertainty analysis – variance The final line of the log is This shows the uncertainty in the model output that is induced by input uncertainties The variance is 73.9 Equal to a standard deviation of 8.6 So although the best estimate of the output is 24.1, the uncertainty in inputs means it could easily be as low as 16 or as high as 33 Estimate of total output variance = 73.9033

17. Exercise

18. GEM-SA course - session 418 A small change Run the same model with Output 11 instead of Output 4 Calculate the coefficient of variation (CV) for this output NB: the CV is defined as the standard deviation divided by the mean

19. More complex analyses

20. GEM-SA course - session 420 Input distributions A normal (gaussian) distribution is generally a more realistic representation of uncertainty Range unbounded More probability in the middle Default is to assume the uncertainty in each input is represented by a uniform distribution Range determined by the range of values found in the input file or separately input

21. GEM-SA course - session 421 Changing input distributions Reopen Project dialog by Project - > Edit … or clicking on Select Options tab Click All unknown, product normal Then OK A new dialog opens to specify means and variances

22. GEM-SA course - session 422 Model 1 example Uniform distributions from input ranges Normal distributions to match Range about 4 std deviations Except for MO Narrower distribution UniformNormal ParameterLowerUpperMeanVariance RESAEREO80200140900 DEFLECT0.610.80.01 FACTOR0.10.50.30.01 MO3010060100 COVER0.60.990.80.01 TREEHT104025100 LAI3.7596.51

23. GEM-SA course - session 423 Effect on UA After running the revised model, we see: It runs faster, with no need to rebuild the emulator The mean is changed a little and variance is halved The emulator fit is unchanged Estimate of mean output is 26.2698, with variance 0.00784475 Estimate of total output variance = 38.1319

24. GEM-SA course - session 424 Reducing MO uncertainty further If we reduce the variance of MO even more, to 49: UA mean changes a little more and variance reduces again Notice also how the emulation uncertainty has increased (0.004 for uniform) This is because the design points cover the new ranges less thoroughly Estimate of mean output is 26.3899, with variance 0.0108792 Estimate of total output variance = 27.1335

25. GEM-SA course - session 425 A homework exercise What happens if we reduce the uncertainty in MO to zero? Two ways to do this Literally set variance to zero Select “Some known, rest product normal” on Project dialog, check the tick box for MO in the mean and variance dialog What changes do you see in the UA?

26. GEM-SA course - session 426 Cross-validation Reopen the Project dialog and select the Options tab Look at the bottom menu box, labelled “Cross- validation” There are 3 options None Leave-one-out Leave final 20% out CV is a way of checking the emulator fit Default is None because CV takes time

27. GEM-SA course - session 427 Cross Validation Root Mean-Squared Error = 0.907869 Cross Validation Root Mean-Squared Relative Error = 4.34773 percent Cross Validation Root Mean-Squared Standardised Error = 1.15273 Largest standardised error is 4.32425 for data point 61 Cross Validation variances range from 0.18814 to 3.92191 Written cross-validation means to file cvpredmeans.txt Written cross-validation variances to file cvpredvars.txt Leave-one-out CV After estimating roughness and other parameters, GEM predicts each training run point using only the remaining n-1 points Results appear in log window Close to 1 (Model 1, output 4, uniform inputs)

28. GEM-SA course - session 428 Leave final 20% out CV This is an even better check, because it tests the emulator on data that have not been used in any way to build it Emulator is built on first 80% of data and used to predict last 20% Standardised error a bit bigger But not bad for just 24 runs predicted Cross Validation Root Mean-Squared Error = 1.46954 Cross Validation Root Mean-Squared Relative Error = 7.4922 percent Cross Validation Root Mean-Squared Standardised Error = 1.73675 Largest standardised error is 5.05527 for data point 22 Cross Validation variances range from 0.277304 to 4.886

29. GEM-SA course - session 429 Output Summary tab The “Output Summary” tab presents all of the key results in a single list Tidier than searching for the details in the log window Although the log window actually has more information Can print using

30. GEM-SA course - session 430 Other options There are various other options associated with the emulator building that we have not dealt with See built in help facility for explanations Also slides at the end of session 3 But we’ve done the main things that should be considered in practice And it’s enough to be going on with!

31. GEM-SA course - session 431 When it all goes wrong How do we know when the emulator is not working? Large roughness parameters Especially ones hitting the limit of 99 Large emulation variance on UA mean Poor CV standardised prediction error Especially when some are extremely large In such cases, see if a larger training set helps Other ideas like transforming output scale A suite of diagnostics is being developed in MUCM See Bastos and O’Hagan on my website http://tonyohagan.co.uk/academic/pub.html Not implemented in GEM-SA yet