1. Eawag: Swiss Federal Institute of Aquatic Science and Technology Analyzing input and structural uncertainty of deterministic models with stochastic, time-dependent parameters Peter Reichert Eawag Dübendorf, ETH Zürich and SAMSI

2. SAMSI meeting Feb. 1, 2007 Contents Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion  Motivation  Approach  Implementation [Tomassini, Reichert, Künsch and Borsuk, 2007, subm.]  Results for a simple epidemiological model [Cintron-Arias, Reichert, Lloyd, Banks, 2007, initiated]  Results for a simple hydrologic model [Reichert and Mieleitner, 2007, in progress]  Discussion

3. SAMSI meeting Feb. 1, 2007 Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

4. SAMSI meeting Feb. 1, 2007 Motivation Problem:  In most applications, there is a bias in results of deterministic models.  This bias is typically caused by input and model structure errors, and not by correlated measurement errors.  Bias modelling describes this bias statistically, but does not directly support the identification of its causes and the formulation of improved model structures. Objective:  Search for improved model structures or stochastic influence factors to model input or parameters that reduce/remove the bias. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

5. SAMSI meeting Feb. 1, 2007 Approach Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

6. SAMSI meeting Feb. 1, 2007 Approach The proposed approach consists of the following steps: 1)Replace selected (constant) parameters of the deterministic model by a stochastic process in time. 2)Estimate the states of this process jointly with the model parameters. 3)Analyze the degree of bias reduction achieved by doing this for different parameters. 4)Analyze the identified time dependence of the parameters for correlation with external influence factors and internal model variables. 5)Improve the deterministic model if such dependences were found. 6)Explain the remaining stochasticity (if any) of the model by including the appropriate stochastic parameter(s) into the model description. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

7. SAMSI meeting Feb. 1, 2007 Implementation Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

8. SAMSI meeting Feb. 1, 2007 Model Deterministc model: Consideration of observation error: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

9. SAMSI meeting Feb. 1, 2007 Model Model with parameter i time dependent: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

10. SAMSI meeting Feb. 1, 2007 Time Dependent Parameter This has the advantage that we can use the analytical solution: The time dependent parameter is modelled by a mean-reverting Ornstein Uhlenbeck process: or, after reparameterization: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

11. SAMSI meeting Feb. 1, 2007 Inference We combine the estimation of  constant model parameters,, with  state estimation of the time-dependent parameter(s),, and with  the estimation of (constant) parameters of the Ornstein-Uhlenbeck process(es) of the time dependent parameter(s),. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

12. SAMSI meeting Feb. 1, 2007 Inference Gibbs sampling for the three different types of parameters. Conditional distributions: Ornstein-Uhlenbeck process (cheap) simulation model (expensive) Ornstein-Uhlenbeck process (cheap) Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

13. SAMSI meeting Feb. 1, 2007 Inference Metropolis-Hastings sampling for each type of parameter: Multivariate normal jump distributions for the parameters  and . This requires one simulation to be performed per suggested new value of . The discretized Ornstein-Uhlenbeck parameter,, is split into subintervals for which OU-process realizations conditional on initial and end points are sampled. This requires the number of subintervals simulations per complete new time series of . Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

14. SAMSI meeting Feb. 1, 2007 Estimation of Hyperparameters by Cross - Validation Due to identifiability problems we select the two hyperparameters (  ) by cross-validation: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

15. SAMSI meeting Feb. 1, 2007 Estimation of Hyperparameters by Cross - Validation For a state-space model of the form we can estimate the pseudo-likelihood from the sample: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

16. SAMSI meeting Feb. 1, 2007 Results for Epidemiological Model Results for a Simple Epidemiological Model  Model  Model Application  Preliminary Results Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

17. SAMSI meeting Feb. 1, 2007 Model A Simple Epidemiological Model: Dushoff et al. 2004 1 S :susceptible I :infected N :„total population“ N-S-I :resistant 2 3 4 5 6 6model parameters 2initial conditions 1standard dev. of meas. err. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

18. SAMSI meeting Feb. 1, 2007 Model Application Influenza Outbreaks in the USA Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

19. SAMSI meeting Feb. 1, 2007 Preliminary Results Model predictions and data Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

20. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis – constant parameters Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

21. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis – beta0 time varying Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

22. SAMSI meeting Feb. 1, 2007 Preliminary Results Time dependent parameter: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

23. SAMSI meeting Feb. 1, 2007 Next Steps  More Fourier terms, more general periodicity.  Cross validation for choice of  and .  Interpretation of remaining stochasticity.  Inclusion of identified forcing in model equations.  Inclusion of stochastic parameter in model equations.  Different data set: measles. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

24. SAMSI meeting Feb. 1, 2007 Preliminary for Hydrologic Model Results for a Simple Hydrologic Model  Model  Model Application  Preliminary Results Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

25. SAMSI meeting Feb. 1, 2007 Model A Simple Hydrologic Watershed Model (1): Kuczera et al. 2006 Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

26. SAMSI meeting Feb. 1, 2007 Model A Simple Hydrologic Watershed Model (2): Kuczera et al. 2006 1 2 3 4 5 6 7 A B 7 model parameters 3 initial conditions 1 standard dev. of meas. err. 3 „modification parameters“ C Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

27. SAMSI meeting Feb. 1, 2007 Model Application Model application:  Data set of Abercrombie watershed, New South Wales, Australia (2770 km 2 ), kindly provided by George Kuczera (Kuczera et al. 2006).  Box-Cox transformation applied to model and data to decrease heteroscedasticity of residuals.  Step function input to account for input data in the form of daily sums of precipitation and potential evapotranspiration.  Daily averaged output to account for output data in the form of daily average discharge. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

28. SAMSI meeting Feb. 1, 2007 Model Application Prior distribution: Estimation of constant parameters: Independent uniform distributions for the loga- rithms of all parameters (7+3+1=11), keeping correction factors (f rain, f pet, f Q ) equal to unity. Estimation of time-dependent parameters: Ornstein-Uhlenbeck process applied to log of the parameter. Hyper-parameters:  = 5d,  fixed, only estimation of initial value and mean (0 for log f rain, f pet, f Q ). Constant parameters as above. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

29. SAMSI meeting Feb. 1, 2007 Preliminary Results Posterior marginals: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

30. SAMSI meeting Feb. 1, 2007 Preliminary Results Max. post. simulation with constant parameters: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

31. SAMSI meeting Feb. 1, 2007 Preliminary Results Residuals of max. post. sim. with const. pars.: Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

32. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis, max. post., constant parameters Residual analysis, max. post., q_gw_max time-dependent Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

33. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis, max. post., s_F time-dependent Residual analysis, max. post., f_rain time-dependent Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

34. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis, max. post., f_pet time-dependent Residual analysis, max. post., f_Q time-dependent Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

35. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis, max. post., k_et time-dependent Residual analysis, max. post., q_latmax time-dependent Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

36. SAMSI meeting Feb. 1, 2007 Preliminary Results Residual analysis, max. post., k_s time-dependent NSparameterNSparameter 0.90f_rain0.79f_pet 0.88s_F0.69q_gwmax 0.87k_s0.67q_latmax 0.83f_Q0.67k_et const. parameter: 0.64 Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

37. SAMSI meeting Feb. 1, 2007 Preliminary Results Time-dependent parameters Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

38. SAMSI meeting Feb. 1, 2007 Preliminary Results Time-dependent parameters Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

39. SAMSI meeting Feb. 1, 2007 Preliminary Results Scatter plot of parameter vs model variables Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

40. SAMSI meeting Feb. 1, 2007 Preliminary Results Scatter plot of parameter vs model variables Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

41. SAMSI meeting Feb. 1, 2007 Preliminary Results Scatter plot of parameter vs model variables Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

42. SAMSI meeting Feb. 1, 2007 Preliminary Results Scatter plot of parameter vs model variables Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

43. SAMSI meeting Feb. 1, 2007 Preliminary Results Scatter plot of parameter vs model variables Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

44. SAMSI meeting Feb. 1, 2007 Next Steps  Search for correlations between time dependent parameters and external influence factors and system variables.  Improve formulation of deterministic model and redo the analysis.  Cross validation for choice of  and .  Inclusion of stochastic parameter in model equations to explain remaining stochasticity. Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

45. SAMSI meeting Feb. 1, 2007 Discussion Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

46. SAMSI meeting Feb. 1, 2007 Discussion 1.Other formulations of time-dependent parameters? 2.Dependence on other factors than time? 3.How to estimate hyperparameters? (Reduction in correlation time always improves the fit.) 4.How to avoid modelling physical processes with the bias term? 5.Learn from more applications. 6.Compare results with methodology by Bayarri et al. (2005). Combine/extend the two methodologies? 7.How to improve efficiency? 8.How to combine statistical with contextual knowledge? 9.? Motivation Approach Implementation Results for epidem. Model Results for hydrol. model Discussion

47. Eawag: Swiss Federal Institute of Aquatic Science and Technology Thank you for your attention