Presentation on theme: "Uncertainty and sensitivity analysis- model and measurements"— Presentation transcript:
1 Uncertainty and sensitivity analysis- model and measurements Marian Scott and Ron Smith and Clive AndersonUniversity of Glasgow/CEH/University of SheffieldGlasgow, Sept 2006
2 Outline of presentation Errors and uncertainties on measurementsSensitivity and uncertainty analysis of modelsQuantifying and apportioning variation in model and data.A Bayesian approachSome general comments
4 The nature of measurement All measurement is subject to uncertaintyAnalytical uncertainty reflects that every time a measurement is made (under identical conditions), the result is different.Sampling uncertainty represents the ‘natural’ variation in the organism within the environment.
5 The error and uncertainty in a measurement The error is a single value, which represents the difference between the measured value and the true valueThe uncertainty is a range of values, and describes the errors which might have been observed were the measurement repeated under IDENTICAL conditionsError (and uncertainty) includes a combination of variance and bias
6 Key properties of any measurement Accuracy refers to the deviation of the measurement from the ‘true’ value (bias)Precision refers to the variation in a series of replicate measurements (obtained under identical conditions) (variance)
7 Accuracy and precision AccurateInaccuratePreciseImprecise
8 Evaluation of accuracy In an inter-laboratory study, known-age material is used to define the ‘true’ ageThe figure shows a measure of accuracy for individual laboratoriesAccuracy is linked to Bias
9 Evaluation of precision Analysis of the instrumentation method to make a single measurement, and the propagation of any errors (theory)Repeat measurements (true replicates) – using homogeneous material, repeatedly subsampling, etc…. (experimental)Precision is linked to Variance (standard deviation)Precision, error, uncertainty, all the terminology again how to estimate
10 The uncertainty rangefor a measurement of 4509 years with quoted error (1 sigma) 20 years, the measurement uncertainty at 2 sigma, would be 4509 40 years or 4469 to 4549 years. We would say that the true age is highly likely to lie within the uncertainty range (95% confidence)
11 The uncertainty range on the mean From the series of 27 replicate measurements made in a single laboratory over a period of several months. The average age of the series is 4497 years. The standard deviation of the series is 30.2 years. The error on the mean is (30.2/27) or 6 years. So the uncertainty (at 2 sigma) on the true age is 4497 12 years or 4485 to 4509 years.
12 Is the quoted error realistic? Commonly judged by making a series of repeat measurements (replicates) and calculating the standard deviation of the series. For the 27 measurements, the st.dev. is 30.2 years but the quoted errors on individual measurements range from 13 to 33 years. So 30 years might be a more realistic individual error.
13 Are two measurements significantly different? Two examples of measurements of a sample. The measurements were made in two different laboratories and so are assumed statistically independent.
14 Case A a) 2759 years 39 and 2811 years20 The difference is -52 years and the error is 44 years, (( )) therefore the uncertainty range is –52 88 years and includes 0.There is no evidence that these two samples do not have the same true age. These two measurements could therefore be legitimately combined in a weighted average .
15 Case B a) 2885 years 37 and 2781years 30. The difference is 104 years and error is 48 years, therefore the uncertainty range is 10496 years or 8 to 200 years and does not include 0.We could conclude that within the individual uncertainties on the measurements, these two samples do not have the same true age. Therefore these two measurements could not be legitimately combined.
16 Can we combine a series of measurements? The results for 6 samples taken from Skara Brae on the Orkney Islands. The samples consisted of single entities (i.e. individual organisms) that represented a relatively short growth interval. The terrestrial samples were either carbonised plant macrofossils (cereal grains or hazelnut shells) or terrestrial mammal bones (cattle or red deer).
17 The test of homogeneity, series of measurements xi, with error siNull hypothesis says measurements are the same (within error)Calculated the weighted mean , xp the test statistic T = (xi –xp)2/si2This should have a 2(n-1) distribution
18 Case A455540, 460540, 452540, 4530 35, 427040, 4735 40Using all 6 measurements, the weighted average is years, and T isT compared with a 2 (5), for which the critical value is 11.07, thus we would reject the hypothesis that the samples all had the same true age, so they cannot be combined.
19 Case B455540, 460540, 452540, 4530 35the weighted average is years, and T is T compared with a 2 (3), for which the critical value is 7.8,thus we would not reject the hypothesis that the samples all had the same true age, and so the weighted average (with its error) could be calculated.
21 uncertainties in input data uncertainty in model parameter valuesConflicting evidence contributes touncertainty about model formuncertainty about validity of assumptions
22 Conceptual system feedbacks Data Model inputs & parameters Policy model results
23 GoalsTransparent approach to facilitate awareness/identification/inclusion of uncertainties within analysisProvide useful/robust/relevant uncertainty assessmentsProvide a means to assess consequences
24 Modelling tools - SA/UA Sensitivity analysis determining the amount and kind of change produced in the model predictions by a change in a model parameter Uncertainty analysis an assessment/quantification of the uncertainties associated with the parameters, the data and the model structure.
25 Modellers conduct SA to determine (a) if a model resembles the system or processes under study,(b) the factors that mostly contribute to the output variability,(c) the model parameters (or parts of the model itself) that are insignificant,(d) if there is some region in the space of input factors for which the model variation is maximum,and(e) if and which (group of) factors interact with each other.
27 Design of the SA experiment Simple factorial designs (one at a time)Factorial designs (including potential interaction terms)Fractional factorial designsImportant difference: design in the context of computer code experiments – random variation due to variation in experimental units does not exist.
28 SA techniques Screening techniques Local/differential analysis O(ne) A(t) T(ime), factorial, fractional factorial designs used to isolate a set of important factorsLocal/differential analysisSampling-based (Monte Carlo) methodsVariance based methodsvariance decomposition of output to compute sensitivity indices
29 Screeningscreening experiments can be used to identify the parameter subset that controls most of the output variability with low computational effort.
30 Screening methodsVary one factor at a time (NOT particularly recommended)Morris OAT design (global)Estimate the main effect of a factor by computing a number r of local measures at different points x1,…,xr in the input space and then average them.Order the input factors
31 Local SALocal SA concentrates on the local impact of the factors on the model. Local SA is usually carried out by computing partial derivatives of the output functions with respect to the input variables.The input parameters are varied in a small interval around a nominal value. The interval is usually the same for all of the variables and is not related to the degree of knowledge of the variables.
32 Global SAGlobal SA apportions the output uncertainty to the uncertainty in the input factors, covering their entire range space.A global method evaluates the effect of xj while all other xi,ij are varied as well.
33 How is a sampling (global) based SA implemented? Step 1:define model, input factors and outputsStep 2:assign p.d.f.’s to input parameters/factors and if necessary covariance structure. DIFFICULTStep 3:simulate realisations from the parameter pdfs to generate a set of model runs giving the set of output values.
34 Choice of sampling method S(imple) or Stratified R(andom) S(ampling)Each input factor sampled independently many times from marginal distbns to create the set of input values (or randomly sampled from joint distbn.)Expensive (relatively) in computational effort if model has many input factors, may not give good coverage of the entire range spaceL(atin) H(ypercube) S(sampling)The range of each input factor is categorised into N equal probability intervals, one observation of each input factor made in each interval.
35 SA -analysisAt the end of the computer experiment, data is of the form (yij, x1i,x2i,….,xni), where x1,..,xn are the realisations of the input factors.Analysis includes regression analysis (on raw and ranked values), standard hypothesis tests of distribution (mean and variance) for subsamples corresponding to given percentiles of x, and Analysis of Variance.
36 Some ‘new’ methods of analysis Measures of importanceVarXi(E(Y|Xj =xj))/Var(Y)HIM(Xj) =yiyi’/NSobol sensitivity indicesFourier Amplitude Sensitivity Test (FAST)
37 How can SA/UA help? SA/UA have a role to play in all modelling stages: We learn about model behaviour and ‘robustness’ to change;We can generate an envelope of ‘outcomes’ and see whether the observations fall within the envelope;We can ‘tune’ the model and identify reasons/causes for differences between model and observations
38 On the other hand - Uncertainty analysis Parameter uncertaintyusually quantified in form of a distribution.Model structural uncertaintymore than one model may be fit, expressed as a prior on model structure.Scenario uncertaintyuncertainty on future conditions.
39 Tools for handling uncertainty Parameter uncertaintyProbability distributions and Sensitivity analysisStructural uncertaintyBayesian frameworkone possibility to define a discrete set of models, other possibility to use a Gaussian process
40 An uncertainty example (1) Wet deposition israinfall ion concentrationRainfall is measured at approximately 4000 locations, map produced by UK Met Office.Rain ion concentrations are measured weekly (now fortnightly or monthly) at around 32 locations.
41 An uncertainty example (2) BUTalmost all measurements are at low altitudesmuch of Britain is uplandAND measurement campaigns showrain increases with altituderain ion concentrations increase with altitudeSeeder rain, falling through feeder rain on hills, scavenges cloud droplets with high pollutant concentrations.
42 An uncertainty example (3) Solutions:More measurementsX at high altitude are not routine and are complicated(b) Derive relationship with altitudeX rain shadow and wind drift (over about 10km down wind) confound any direct altitude relationships(c) Derive relationship from rainfall map model rainfall in 2 separate components
44 An uncertainty example (5) Wet deposition is modelled byr actual rainfalls rainfall on ‘low’ ground (r = s on ‘low’ ground, and(r-s) is excess rainfall caused by the hill)c rain ion concentration as measured on ‘low’ groundf enhancement factor (ratio of rain ion concentrationin excess rainfall to rain ion concentration in‘low’ground rainfall)deposition = s.c + (r-s).c.f
45 An uncertainty example (6) RainfallConcentrationDeposition
46 An uncertainty example (7) r modelled rainfall to 5km squares provided by UKMO - unknown uncertaintyscale issue - rainfall a point measurementmeasurement issue - rain gauges difficult touse at high altitudeoptimistic 30% pessimistic 50%how is the uncertainty represented?(not e.g. 30% everywhere)
47 An uncertainty example (8) s some sort of smoothed surface(change in prevalence of westerly windsmeans it alters between years)c kriged interpolation of annualrainfall weighted mean concentrations(variogram not well specified)assume 90% of observations within ±10% of correct valuef campaign measurements indicate valuesbetween 1.5 and 3.5
48 An uncertainty example (9) Output measures in the sensitivity analysis are the average flux (kg S ha-1 y-1) for(a) GB, and(b) 3 sample areas
49 An uncertainty example (10) Morris indices are one way of determining which effects are more important than others, so reducing further work.but different parameters are important in different areas
50 An uncertainty example (11) 100 simulations Latin Hypercube Sampling of 3 uncertainty factors:enhancement ratio% error in rainfall map% error in concentration
51 An uncertainty example (12) Note skewed distributions for GB and for the 3 selected areas
52 An uncertainty example (13) Mean of 100 simulationsStandard deviationOriginal
53 An uncertainty example (14) CV from 100 simulationsPossible bias from 100 simulations
54 An uncertainty example (15) model sensitivity analysis identifies weak areaslack of knowledge of accuracy of inputs a significant problemthere may be biases in the model output which, although probably small in this case, may be important for critical loads
55 Conclusions so far The world is rich and varied in its complexity Modelling is an uncertain activitySA/UA are an important tools in model assessmentThe setting of the problem in a unified Bayesian framework allows all the sources of uncertainty to be quantified, so a fuller assessment to be performed.
56 Bayesian Approach to Model Uncertainty, Calibration, Sensitivity Analysis ….
57 Bayes Essentials Eg experimental determination of a constant Data Posterior ideasaboutPrior ideasabout
58 Bayes’ Rule Bayes Essentials likelihood – from model for data generation
59 + General form: Bayes Essentials a (statistical) model describing data generation, specified in a likelihoodObservations+UnknownFor inferences to be coherent they must work in this way.
60 Computer/Numerical Models Scientific understanding of environmental processes often expressed in a computer/numerical model …
61 Sheffield Dynamic Global Vegetation Model, SDGVM Computer/Numerical ModelsSheffield Dynamic Global Vegetation Model, SDGVMClimateCO2, NSoilPHYSIOLOGYBIOPHYSICSWATER & NUTRIENTFLUXESPLANTSTRUCTURE&PHENOLOGYDISTURBANCEVEGETATIONDYNAMICS
62 CO2: emissions vs atmospheric increase Computer/Numerical ModelsCO2: emissions vs atmospheric increase‘Sinks for Anthropogenic Carbon’, Physics Today 2002, J L Sarmiento & N Gruber
63 Computer/Numerical Models usually deterministic, always wronghow to quantify the uncertainty?
64 Statistical Viewpoint on Numerical Models INPUTOUTPUTUncertain as a representation of reality:may not be knownmay be inadequate— uncertainty analysis— model inadequacy
65 Statistical Viewpoint on Numerical Models EmulationNumerical model: a function mapping inputs into outputsOutputInputxIf model outputs available only at a limited number of inputs?How represent knowledge about the model?
66 Bayes Formulation Statistical Viewpoint on Numerical Models Put a distribution on the space of possible functions;ie, treat as randomand use the Bayes machinery to update knowledge about it from runs of the computer model/simulator.(Bayes rule!)The probability distribution ofcalled an emulator
67 Numerical Models and Reality - Calibration, Model Inadequacy, Predictive UncertaintyMain goal of modelling: to learn about reality.Relation of numerical model to reality: represent via a statistical model and use the inference machinery to learn about it.One formulation:observations,the true process,the numerical modelobservational errorregression parametermodel inadequacy
68 Treat also as an unknown function Earlier, used runs of numerical model to learn about and build emulator.Now in same way use observed data and the emulator to learn aboutvia Bayes rule
69 Calibration: using observed data to learn about model inputs . Findparameters of the two GPsvia Bayes ruleCan integrate out and use maximizing to getsummarizing information about
70 Prediction and predictive uncertainty: ie what is ?Conditionally is a Gaussian processCombine withfor inference aboutand further combine withHence predictions and their uncertainty.
71 GEM software (Gaussian Emulation Machine) GEM-SA, GEM-CALGenerates a statistical emulator of a computer code from training data consisting of an arbitrary set of inputs and the resulting outputs.Gives the following:prediction of code output at any untried inputs, taking account of uncertainty in one or more of the code inputs.main effects of each individual input.joint effects of each pair of inputs.percentage allocation to the variance from each individual input.Calibrates code to observations, quantifies model inadequacy & predictive uncertainty
72 Some References: For GEM software see www.ctcd.shef.ac.uk Kennedy, M. C. & O’Hagan, A. (2001) Bayesian calibration of computer models J. Roy. Statist. Soc. B, 63,Kennedy, M. C., O’Hagan, A. & Higgins, N. (2002) Bayesian analysis of computer code outputs. In Quantitative Methods for Current Environmental Issues, eds CW Anderson, V Barnett, P Chatwin & AH El-Shaarawi. Springer, London.Oakley, J. E. & O’Hagan, A. (2004) Probabilistic sensitivity analysis. J. Roy. Statist. Soc. B, ,Saltelli A, Chan K, Scott E M (2000) Sensitivity Analysis. Wiley.Royal Society of Chemistry, Analytical Methods Sub-committee (web)