1. Keith Beven Lancaster University, UK
Assessment of uncertainty of environmental models based on the equifinality thesis
Keith Beven
Lancaster University, UK

2. A Manifesto for the Equifinality Thesis
One aim of environmental models is to achieve a single true description of governing processes (reality) - especially for predicting impacts of change
Difficult to achieve in applications to places that are unique in their characteristics and where (nonlinear) predictions are subject to input, observation, and model structural errors
There may instead be many descriptions that are compatible with current understanding and available observations
The concept of the single description may remain a philosophical axiom or theoretical aim but is impossible to achieve in practice
So we must accept that there may be many feasible descriptions, or a concept of equifinality, as the basis for a new approach

3. A Manifesto for the Equifinality Thesis
So what if we accept multiple possible models and parameter sets…… <M(θ)1, M(θ)2, … M(θ)N > ???
There is no optimum model - can only assess the likelihood of a model being acceptable
Different acceptable models will produce different predictions - especially of impacts of future change
Should therefore assess the resulting uncertainty in predictions
Should be prepared to revise predictions as new data become available
Models should be rejected if shown to be non-behavioural (hypothesis testing by model rejection?)

4. Equifinality: an empirical result
Fitting van Genuchten parameters in modelling recharge
after Binley and Beven, Groundwater, 2002

5. Equifinality: an empirical result
Fitting parameters of the MAGIC geochemistry model
after Page et al., Water, Soil and Air Pollution, 2003

6. Equifinality: an empirical result
Fitting parameters of the Penman-Monteith equation in predicting patch scale latent and sensible heat fluxes
after Schulz and Beven, Hydrological Processes, 2003

7. Uncertainty is not only Statistics
Classical approach to uncertainty of treating total model error in linear (or log linear) form
O(X, t) = Ô{M(θ,β)} + ε(X,t)
Implicit assumption that the model is correct (or can be made to be correct through another additive term) and that any error in the inputs and other boundary conditions cascades through the model linearly
Full power of linear statistics can then be used to estimate
L[O(X,t)|θ]
and posterior distribution of parameters, θ

8. Uncertainty is not only Statistics
But environmental models are generally nonlinear, are subject to important uncertainties in input and boundary conditions and (we suspect) suffer from model structural error
More interested in L[M(θ)|O(X,t)]
Additive total error may be difficult to justify but may be an approximation that will often be useful
BUT…… from the single total error series we cannot easily deconstruct the sources of uncertainty in the modelling process

9. Sources of subjectivity in assessing model error
We like to think that we are objective and scientific in modelling but this is not entirely the case. There are always choices to be made...
Choice of models to be considered (including processes to be included, closure and boundary conditions),
Choice of ranges (prior distributions) of parameter values to be considered
Choice of input data (and input data errors) with which to drive the model (including future scenarios for prediction)
Choice of error model, performance measure(s) or allowable error to be considered acceptable (taking account of scale, heterogeneity and incommensurability effects)

10. Equifinality and the Modelling Process
Take a (thoughtful) sample of all possible models (structures + parameter sets)
Evaluate those models in terms of both understanding and observations in a particular application
Reject those models that are non-behavioural (but note that there may be a scale problem in comparing model predictions and observations)
Devise testable hypotheses to allow further models to be rejected
[If all models rejected, revise model structures……]
This is the essence of the GLUE methodology
(Bayesian priors and likelihood functions based on additive error as special case where strong assumptions justified)

11. Published Applications of GLUE
Rainfall-runoff modelling
Stochastic rainfall models
Radar Rainfall calibration
Modelling catchment geochemistry
Modelling flood frequency and flood inundation
Modelling river dispersion
Modelling soil erosion
Modelling land surface to atmosphere fluxes
Modelling atmospheric deposition and critical loads
Modelling groundwater capture zones
Modelling groundwater recharge
Modelling water stress cavitation and tree death
Modelling forest fires

12. Generalised Likelihood Uncertainty Estimation (GLUE)
Note 1: it is the parameter set in combination with the given input and boundary condition data that gives a behavioural model - complex parameter interactions may mean that marginal distributions and global covariances have little relevance
Note 2: implicit treatment of complex errors in likelihood weighting of simulations (effectively assuming that prediction errors for any behavioural model in prediction will be “similar” to conditioning periods)
Note 3: prediction limits are conditional on choices
Note 4: depends on having sufficient sample to find upper limit of performance

13. Deconstructing total model error
Extended GLUE methodology
insist on model providing predictions within range of “effective observation error” of evaluation variables
effective observation error constructed to take account of scale dependencies and incommensurability (may be dependent on model implementation)
models providing predictions outside range are rejected as non-behavioural (multiple models can be included in same formalism but possible that all models may be rejected)
success may depend on allowing realisations of error in input and boundary condition data

14. Incorporating Observational Errors into GLUE
Initial condition errors generally poorly known and are difficult to assess (no means of direct measurement) but effect will die out as simulation proceeds (eventually)
Input and boundary condition data errors generally poorly known because of measurement technique limitations and will be processed nonlinearly through the model
Model structural error is ubiquitous and difficult to assess - claims to physical realism do not exclude possibility of structural error but want models that are consistent with effective observational error
Effective observational error may be difficult to assess because of scale, heterogeneity and incommensurability effects - but provides a critical limit for model rejection

15. Predictive distribution over all behavioural models: (A) predictions encompass new observation

16. Predictive distribution over all behavioural models: (B) predictions do not encompass new observation
………But models are still behavioural
(or can be rejected on basis of new observations)

17. Bukmoongol Catchment – Location

18. Bukmoongol – Stage/Discharge Errors
Concrete Weir has three
sharp-crested rectangle
sections – 2 only active at
peak discharges
+/- 20%

19. Rainfall Magnitude Errors
Event 1
Event 2
Event 3

20. Results – Model Structural Error?
Recession
Error
IC errors
Log(Discharge)
Timing Errors
And under prediction
Standard Z Value
(Scaled PM2)

21. Future prospects
A possible way forward – To think about moving from evaluations that are based on ‘lumped’ additive error terms to a framework that considers error source terms individually
Generic technique, simple to implement in an uncertainty framework (GLUE) and based on explicit assumptions– even if assumptions may be imprecise (e.g. “effective measurement error”)
Allows for equifinality of model structures, input realisations and parameter sets in producing behavioural models - interaction of input errors and model structure may be important
Allows for evaluation of model structures by rejection (hypothesis testing) (see philosophy paper, Proc. Roy. Soc., 2002)

22. Future prospects: model structural error
But what if these tests reveal that all models tried are non- behavioural and should be rejected?
?? Ensure that model space has been searched adequately
?? Relax rejection criteria
?? Add a model inadequacy term to compensate for structural error - but will be time variable and will have additional parameters for each model realisation, and how much correction should be allowed?
?? Find a better model - uncertainty estimation should NOT remove the need for creative and constructive modelling

23. and if you might possibly still want to read more…...
Beven, K J, 2000, Uniqueness of place and process representations in hydrological modelling, Hydrology and Earth System Sciences, 4(2),
Beven, K. J., 2002, Towards an alternative blueprint for a physically-based digitally simulated hydrologic response modelling system, Hydrol. Process., 16(2),
Beven, K. J., 2002, Towards a coherent philosophy for environmental modelling, Proc. Roy. Soc. Lond., A458, (comment by Philippe Baveye and reply still to appear)
Beven, K J and Young, P C, 2003, Comment on Bayesian Recursive Parameter Estimation for Hydrologic Models by M Thiemann et al. Water Resources Research, 39(5), doi: /2001WR001183