Course round-up subtitle- Statistical model building Marian Scott University of Glasgow Glasgow, Aug 2012

Outline of presentation Statistical models- what are the principles – describing variation – Empiricism Knowing your data Fitting models- calibration Testing models- validation or verification and in general inference- what matters? Quantifying and apportioning variation in model and data. Stochastic and deterministic models. intro to uncertainty and sensitivity analysis resources

Step 1 why do you want to build a model- what is your objective? what data are available and how were they collected? is there a natural response or outcome and other explanatory variables or covariates?

Modelling objectives explore relationships make predictions improve understanding test hypotheses

Conceptual system Data Model Policy inputs & parameters model results feedbacks

Why model? Purposes of modelling: – Describe/summarise – Predict - what if…. – Test hypotheses – Manage What is a good model? – Simple, realistic, efficient, reliable, valid

Value judgements Different criteria of unequal importance key comparison often comparison to observational data (RSS, AIC......) but such comparisons must include the model uncertainties and the uncertainties on the observational data.

Questions we ask about models Is the model valid? Are the assumptions reasonable? Does the model make sense based on best scientific knowledge? Is the model credible? Do the model predictions match the observed data? How uncertain are the results?

Stages in modelling Design and conceptualisation: – Visualisation of structure – Identification of processes – Choice of parameterisation Fitting and assessment – parameter estimation (calibration) – Goodness of fit

a visual model- atmospheric flux of pollutants Atmospheric pollutants dispersed over Europe In the 1970 considerable environmental damage caused by acid rain International action Development of EMEP programme, models and measurements

The mathematical flux model L: Monin-Obukhov length u*: Friction velocity of wind c p : constant (=1.01) : constant (=1246 gm -3 ) T: air temperature (in Kelvin) k: constant (=0.41) g: gravitational force (=9.81m/s) H: the rate of heat transfer per unit area gasht: Current height that measurements are taken at. d: zero plane displacement

what would a statistician do if confronted with this problem? Look at the data understand the measurement processes think about how the scientific knowledge, conceptual model relates to what we have measured

Step 2- understand your data study your data learn its properties tools- graphical

The data- variation soil or sediment samples taken side-by-side, from different parts of the same plant, or from different animals in the same environment, exhibit different activity densities of a given radionuclide. The distribution of values observed will provide an estimate of the variability inherent in the population of samples that, theoretically, could be taken.

Activity (log 10 ) of particles (Bq Cs-137) with Normal or Gaussian density superimposed Variation

measured atmospheric fluxes for 1997 measured fluxes for 1997 are still noisy. Is there a statistical signal and at what timescale?

Key properties of any measurement Accuracy refers to the deviation of the measurement from the true value Precision refers to the variation in a series of replicate measurements (obtained under identical conditions)

Accurate Imprecise Inaccurate Precise Accuracy and precision

Evaluation of precision Analysis of the instrumentation method to make a single measurement, and the propagation of any errors Repeat measurements (true replicates) – using homogeneous material, repeatedly subsampling, etc…. Precision is linked to Variance (standard deviation)

The nature of measurement All measurement is subject to uncertainty Analytical uncertainty reflects that every time a measurement is made (under identical conditions), the result is different. Sampling uncertainty represents the natural variation in the organism within the environment.

The error and uncertainty in a measurement The error is a single value, which represents the difference between the measured value and the true value The uncertainty is a range of values, and describes the errors which might have been observed were the measurement repeated under IDENTICAL conditions Error (and uncertainty) includes a combination of variance and bias

Effect of uncertainties Lack of observations contribute to – uncertainties in input data – uncertainty in model parameter values Conflicting evidence contributes to – uncertainty about model form – uncertainty about validity of assumptions

Data properties Nature and distribution of the data- continuous, counts.... Normal, exponential, poisson, maybe need a transformation Missing data- outliers- limits of detection Use pictures to explore

Step 3- build the statistical model Outcomes or Responses Causes or Explanations these are the conditions or environment within which the outcomes or responses have been observed -the covariates. This has very much been the focus of much of the week- whether a linear model, a smooth flexible model, a time series model, a bayesian model.....

Statistical models In experiments many of the covariates have been determined by the experimenter but some may be aspects that the experimenter has no control over but that are relevant to the outcomes or responses. In observational studies, these are usually not under the control of the experimenter but are recorded as possible explanations of the outcomes or responses. Some of your experiments are in the lab where you control factors such as temperature, concentration etc, some are in the world laboratory where you observe...

Specifying a statistical model Models specify the way in which outcomes and causes link together, eg. Metabolite ~ Temperature The ~ sign does not indicate equality in a mathematical sense and there should be an additional item on the right hand side giving a formula:- Metabolite ~ Temperature + Error Thinking also about possible random effect models (see Marks section)

Specifying a statistical models In mathematical terms, there will be some unknown parameters to be estimated, and some assumptions will be made about the error distribution Metabolite = + temperature + ~ N(0, σ 2 ) All statistical models make assumptions and we need to check them

Are you a bayesian? What does that mean? It means, you have prior information (belief) that you want to include in your statistical model You need to find a way of capturing this in the prior distribution Model output then a posterior distribution on the quantity of interest- automatically incorporates uncertainty

Model calibration Statisticians tend to talk about model fitting, calibration means something else to them. Methods- least squares or maximum likelihood least squares:- find the parameter estimates that minimise the sum of squares (SS) SS= (observed y- model fitted y )2 maximum likelihood- find the parameter estimates that maximise the likelihood of the data

Calibration-using the data A good idea, if possible to have a training and a test set of data-split the data (90%/10%) Fit the model using the training set, evaluate the model using the test set. why? because if we assess how well the model performs on the data that were used to fit it, then we are being over optimistic other methods: bootstrap and jackknife

Model validation what is validation? Fit the model using the training set, evaluate the model using the test set. why? because if we assess how well the model performs on the data that were used to fit it, then we are being over optimistic other methods: bootstrap and jackknife

Which variables to include Use your science knowledge Use pictures to look for patterns Maybe use some of the more algorithmic ways to select the set (stepwise, BSR...) How to compare models? Nested models (ANOVA, likelihood ratio test)

More than 1 model- how well should models agree? 6 ocean models (process based-transport, sedimentary processes, numerical solution scheme, grid size) used to predict the dispersal of a pollutant Results to be used to determine a remediation policy for an illegal dumping of radioactive waste The what if scenario investigation The models differ in their detail and also in their spatial scale

Predictions of levels of cobalt-60 Different models, same input data Predictions vary by considerable margins Magnitude of variation a function of spatial distribution of sites Another component of uncertainty

Statistical models and process models Loch Leven, modelling nutrients process model based on differential equations statistical model based on empirically determined relationships We can connect the two but this is a research area rather than a routine...

Uncertainty and sensitivity analysis

Uncertainty (in variables, models, parameters, data) what are uncertainty and sensitivity analyses?

Effect of uncertainties Lack of observations contribute to – uncertainties in input data – uncertainty in model parameter values Conflicting evidence contributes to – uncertainty about model form – uncertainty about validity of assumptions

Modelling tools - SA/UA Sensitivity analysis determining the amount and kind of change produced in the model predictions by a change in a model parameter Uncertainty analysis an assessment/quantification of the uncertainties associated with the parameters, the data and the model structure.

Modellers conduct SA to determine (a)if a model resembles the system or processes under study, (b)the factors that mostly contribute to the output variability, (c)the model parameters (or parts of the model itself) that are insignificant, (d)if there is some region in the space of input factors for which the model variation is maximum, and (e)if and which (group of) factors interact with each other.

SA flow chart ( Saltelli, Chan and Scott, 2000)

Design of the SA experiment Simple factorial designs (one at a time) Factorial designs (including potential interaction terms) Fractional factorial designs Important difference: design in the context of computer code experiments – random variation due to variation in experimental units does not exist.

Global SA Global SA apportions the output uncertainty to the uncertainty in the input factors, covering their entire range space. A global method evaluates the effect of x j while all other x i,i j are varied as well.

How is a sampling (global) based SA implemented? Step 1:define model, input factors and outputs Step 2:assign p.d.f.s to input parameters/factors and if necessary covariance structure. DIFFICULT Step 3:simulate realisations from the parameter pdfs to generate a set of model runs giving the set of output values.

SA -analysis At the end of the computer experiment, data is of the form (y ij, x 1i,x 2i,….,x ni ), where x 1,..,x n are the realisations of the input factors. Analysis includes regression analysis (on raw and ranked values), standard hypothesis tests of distribution (mean and variance) for subsamples corresponding to given percentiles of x, and Analysis of Variance.

How can SA/UA help? SA/UA have a role to play in all modelling stages: – We learn about model behaviour and robustness to change; – We can generate an envelope of outcomes and see whether the observations fall within the envelope; – We can tune the model and identify reasons/causes for differences between model and observations

On the other hand - Uncertainty analysis Parameter uncertainty – usually quantified in form of a distribution. Model structural uncertainty – more than one model may be fit, expressed as a prior on model structure. Scenario uncertainty – uncertainty on future conditions.

Tools for handling uncertainty Parameter uncertainty – Probability distributions and Sensitivity analysis Structural uncertainty – Bayesian framework one possibility to define a discrete set of models, other possibility to use a Gaussian process – model averaging

An uncertainty example ( Ron Smith ) Original Mean of 100 simulations Standard deviation

An uncertainty example CV from 100 simulations Possible bias from 100 simulations

An uncertainty example model sensitivity analysis identifies weak areas lack of knowledge of accuracy of inputs a significant problem there may be biases in the model output which, although probably small in this case, may be important for critical loads

Take home message Only able to give you a flavour of what might be possible Good environmental science and good statistical science is key for all problems Think critically- test and re-test your hypotheses and assumptions

Take home message Resources Many good books (have seen some of these over the sessions- not one size fits all JISC mail list- Envstat (worth joining) Royal Statistical Society has an Environmental Statistics section, sometimes holds tutorial meetings on topics.