1. Over-parameterisation and model reduction Neil Crout Environmental Science School of Biosciences University of Nottingham

2. Co-workers Davide Tarsitano Glen Cox James Gibbons Andy Wood Jim Craigon Steve Ramsden Yan Jiao Tim Reid with thanks to BBSRC and Leverhulme

3. Motivation

4. Increase the complexity of the model to improve agreement with observations Can continue this until ALL the variation in the data is accounted for – Including the noise! Models can become overparameterised Over-parameterisation

5. Judgements have to be made about the level of complexity in a model (parsimony) In a statistical context methods for model selection exist Key feature - comparing models Over-parameterisation

6. So what! Our models are not statistical Mechanistic No fitting involved Predicting lots of different things (multi-variate) Have to account for lots of different cases Did we mention they’re mechanistic – Newton – Thermodynamics – etc all on our side

7. Overparameterisation and mechanistic models? Mechanistically inspired – but empirically implemented So normal statistical rules of business apply Model parameters are often not formally fitted – worse they may be ‘tuned’ – might be measured (but model structure may not be consistent with real value) Often (maybe always) models are implicitly fitted – comparisons are made with the real world – model changes – development iterates But normal rules of business apply – greater complexity may not equal greater realism

8. Overparameterisation and mechanistic models? Mechanistic models have the potential for over- parameterisation This is a source of uncertainty Perhaps of equal importance Mechanistic models are cited as tools to test understanding For this we do need to know whether a model’s formulation really can be supported by observations

9. Judgements have to be made about the level of complexity in a model Mostly this is subjective – supported by tools such as sensitivity analysis etc – sometimes by consideration of alternative model formulations – but this is generally difficult…. Want some statistical support for model selection But that requires a set of alternative models to compare Model Reduction – Why?

10. Want to assess whether a model has the most appropriate level of detail We can get some idea on this by comparing models of the same system which have different levels of detail But, we don’t have lots of different models So we consider reducing the model we have to create alternative model formulations which can be compared Comparison is restricted – reduced models are drawn from the same source – but hopefully better than no comparison at all

11. Some illustrations

12. Example: radioacaesium uptake

13. ‘Absalom’ radiocaesium uptake model NH 4 % clay % C K+K+ pH θ clay θ humus K x soil M_camgCEC humus CEC clay K x humus (2) mK (1) Kd clay (1) Kd humus RIP clay (2) CF (2) KdlKdr D factorCs sol Cs soil Cs plant

14. Model Reduction – How? Start with a ‘full’ model of a system Reduce it (systematically, automatically) Producing a ‘set’ of alternative model formulations for the same system Reduction? – Inter-connected nature of typical models mean can’t simply leave things out – We replace variables with a constant – e.g. the mean or median that the variable attains in the full model

15. Reduction by variable replacement Suppose the full model has a variable V i Which is a function of the model variables, inputs, and parameters Arrange so that the model has Where  is either 0 (no replacement) or 1 (replaced) Systematically reducing the model then involves setting the values of the vector 

16. Search the model replacement space Search the combination of possible models (i.e. varying  ) – exhaustive if computationally feasible – stochastic search (e.g. MCMC) can be more efficient – discrete space Calculate a posterior probability for each model configuration This is based on comparison with observations – So this is a data-led approach – Data has to be representative for analysis to be representative – Certainly need to be aware of the data’s limitations

17. 10 candidate variables – 2 10 combinations NH 4 % clay % C K+K+ pH θ clay θ humus K x soil M_camgCEC humus CEC clay K x humus (2) mK (1) Kd clay (1) Kd humus RIP clay (2) CF (2) KdlKdr D factorCs sol Cs soil Cs plant

18. Posterior probability for top 10 models Full Model

19. NH 4 % clay % C K+K+ pH θ clay θ humus K x soil M_camgCEC humus CEC clay K x humus (2) mK (1) Kd clay (1) Kd humus RIP clay (2) CF (2) KdlKdr D factorCs sol Cs soil Cs plant Full model (as published)

20. NH 4 % clay % C K+K+ pH θ clay θ humus K x soil M_camgCEC humus CEC clay K x humus (2) mK (1) Kd clay (1) Kd humus RIP clay (2) CF (2) KdlKdr D factorCs sol Cs soil Cs plant this one works better…

21. Better? Reduced uncertainty, better predicting models – Tested with independent data Requiring fewer inputs – Specifically soil pH – Model is used spatially, so this is a good thing operationally Greater confidence that the model is not over- parameterised….? Diagnostic to guide/support model development

22. Methane Emissions from Wetlands

23. Methane emission from wetlands

24. Top ‘performing’ models Full Model

25. Mechanistic Interpretation Model diagnostics – present results as probability of replacing specific variables Summing over the model combinations explored – seasonal transfer of plant matter to soil organic matter (100%) – reducing methane oxidation from mechalis-menten to first order (39%) – temperature dependency of methane oxidation (15%) – seasonal dependency of plant growth (99%).

26. C & N dynamics in the arctic N15 applied to field plots 3 years subsequent measurement used to parameterise a model based on MBL-GEM

27. C & N dynamics in the arctic

28. 14 candidate variables (16k combinations)

29. Ensemble prediction over set of models

30. Sirius – Wheat Crop Growth Model

31. Soil moisture Q Qfc Qwp exw[1] aw[1] uw[1] AW[1] Water balance Si leafarea Percolation WP (Rain +irrigation) si Waterbalance.Si AW[1..30] exw[1..30] SOIL.EXW[1..30] aw[1..30] Nex[1..30] x Equilibrium dn Nuw[1..30] Naw[1..30] aw[1..30] uw[1..30] exw[1..30] Nex[1..30] Soil evap EVAPO.evsoil evsoil aw[1..5] exw[1..5] GRAIN GrainDemand DeadLeafN PoolN GrainN AvN SoilN CropN UptakeN leafarea LeafNc ExLeafNc ExStemNc StemNc MinGrainDemand GrainDM TTOUT Biomass BIOANTH Demand BiomassBGF GrainNoverGrainDm AreaANTH N’Pulses Qr Norganic Ndp Ni Nf Nm Ts Ta x[1..8] Naw[1] SumP uw[1..8] aw[1..8] Naw[1..8] Nuw[1..8] Q Qwp Qfc Rain+Irr Fert Tmean Q Wateruptake aw[1..30] x WP(Pottrans) EVAPO.pottrans exw[1..30] RZexw TransP SF Rootlen si CropN AvN stembiomass Nex[1..30] StemN anforP Naw[1..30] leafarea biomass minstemDemand UptakeN CropN leafarea StemNc LeafNc leafarea StemExN Leafdemand MaxStemdemand SF Rootlen Dry matter PAR rad Biomass Tau drfacgrowth RUE EarWt SF Anthesis Therm Lastleaf PHENO Grainend Lastleaf WINTER CROPUP Anthesis Grainstart Therm IPHASE TTOUT BIOANTH Biomass BiomassBGF AreaANTH Reduc FleafNo Vernalisation Potlfno PrimordNo Amnlfno Vprog Roots TTROOT Rootlength TTOUT Anthesis CANOPY SHOUT TTOPT CanTemp LAIStage Areamax TTFIX Areaopt Deadleaves TTCAN Maxgaklir GAKLIR Leafarea TTOUT Soilmax SoilMin TempMin TempMax Leafnumber DrFacLAI TTOUT SF Reduc EVAPO SOIL.AW[1..30] PTSOIL AVWATER EVSOIL SLOSL ECUM PTAY RN Def WATCON TAU tadj ENAV EVAP PENMAN HSLOP(Tmean) aw[1..30 TCmin TCMax Hsoil Soilmax Alpha exw[1..30] PotTrans leafarea Transp Hcrop CONDUC Tmean Temp Max Temp Min Soilmin TDeepsoil Rad Rootlen DrFacgrowth DrFacLai SF Inputs Temp Max Rain Radiation TTOUT TDeepsoil VARIETY Soil Results

32. Top 75% of the distribution/ensemble

33. 40 other variables <<50% Interpretation.... Crop N physiology 50% Vernalisation 99% N-mineralisation Temp/moisture adjustment~99% Canopy Temperature 96% Diurnal adjustment of thermal time 99% Nitrogen Leaching 50%

34. Other applications… Applying the same/similar approach to other models, e.g. Marine Ecology Models (carbon cycling) JULES – land surface scheme for GCM FARM-ADAPT – very large farm management model – Linear programming based model – V. large number of variables (10 000s) – More challenging

35. Some conclusions Whether a model’s formulation is appropriate is uncertain Model reduction provides a way to test (challenge) model formulation (perhaps a brutal test) May give us some insurance against over- parameterisation (reduce uncertainty) Does test the understanding built into our models All models investigated so far can be reduced by variable replacement Most reduced models have some advantage

36. Useful references? NMJ Crout, D Tarsitano, AT Wood. Is my model too complex? Evaluating model formulation using model reduction. Environmental Modelling & Software, in press. JM Gibbons, GM Cox, ATA Wood, J Craigon, SJ Ramsden, D Tarsitano, NMJ Crout (2008). Applying Bayesian Model Averaging to mechanistic models: an example and comparison of methods. Environmental Modelling & Software, 23:973-985. Cox GM, Gibbons JM, Wood ATA, Craigon J, Ramsden SJ, Crout NMJ (2006). Towards the systematic simplification of mechanistic models. Ecological Modelling 198:240-246. Bernhardt, K. 2008. Finding alternatives and reduced formulations for process based models. Evolutionary Computation 16:1-16. Alternative approach. Asgharbeygi, N., Langley, P., Bay, S., Arrigo, 2006. Inductive revision of quantitative process models. Ecol. Mod., 194:70-79. Alternative approach to model reduction, albeit without a statistical framework Brooks RJ, Tobias AM (1996) Choosing the best model: level of detail, complexity, & model... Mathl. Comput. Mod. 24:1-14. Interesting article on model utility Anderson, T.R., 2005. Plankton functional type modelling: running before we can walk? J. Plankton Research 27:1073-1081. Nice discussion of model complexity in context of marine systems. Quite controversial, replies to the replies to the replies. Jakeman, A.J, Letcher, R.A., Norton, J.P., 2006 Ten iterative steps in development and evaluation of environmental models. Environmental Modelling & Software 21, 602- 614. Very nice discussion on what is good practice in model development Myung, J., Pitt, M. A., 2002. When a good fit can be bad. Trends. Cogn. Sci., 6:421- 425. Good introduction to model selection criteria Burnham, K.P., Anderson, D.R., 2002. Model Selection and Multimodel Inference: A Practical Information-Theoretical Approach. 2nd ed. New York: Springer-Verlag. Ultimate reference, but not without critics.