1. Model performance & Evaluation Calibration and Validation

2. Calibration and validation
Prepared by: Ilyas Masih and Marloes Mul
UNESCO-IHE, The Netherlands

3. Evaluation of model performance
The process of assessing the performance of a model requires the subjective and/or objective estimates of the “closeness” of the simulated behaviour of the model to observations.
Qualitative or subjective assessment: to assess the systematic (e.g., over- or under prediction) and dynamic (e.g., timing, rising limb, falling limb, and base flow) behaviour of the model.
Objective assessment: requires the use of a mathematical estimate of the error between the simulated and observed hydrologic variable(s) – i.e. objective or efficiency criteria.

4. Evaluation of model performance
The confidence in use of a model (for reasoning and the formulation of testable predictions) will depend on how ‘close’ the model is to reality.
degree of accuracy (bias), degree of precision (uncertainty) and degree of correspondence (‘sameness’ of the quantities in question) in the comparative evaluation of both fields (observation vs simulation).
Calibration is a process of changing the model input values (usually parameters) to match as closely as possible the simulated behaviour of the model to the observation.

5. Schematic of model evaluation and Calibration process
Vrugt (2004)

6. Calibration Improving the model results by
Changing parameters
For when the understanding of the processes is insufficient
For when the information is insufficient to determine the parameters

7. Parameter Obtained by measuring
Derived from observations (such as the K-value)
Derived by calibration

8. Commonly used variables against which water systems models are evaluated and calibrated
Water balance components: e.g. streamflows, groundwater levels, actual evapotranspiration, soil moisture
Water quality: e.g. sediment, Nitrogen, Phosphorous
Water allocation: demand and supply
Other? (e.g. reservoir volume/level and releases)

9. Commonly used Objective functions (criteria) in water systems models’ evaluation and calibration
Bias
Mean square Error; root mean square error, relative root mean square error
Coefficient of determination
Nash-Sutcliffe efficiency
Other? (e.g. comparison of the slopes of the observed and simulated flow duration curves)

10. Objective functions: example Nash-Stucliffe efficiency (NSE), coefficient of determination (R2) and Bias(error) (%)

11. Calibration approaches: Manual calibration
Most widely used calibration method
Visual comparison of measured and simulated data
Semi-intuìtive trial and error process for parameter adjustment
Excellent model calibrations, but manual calibration...
is highly labor-intensive (human resources)
is difficult to learn
procedures are model-dependent
results are user-dependent

12. Calibration approaches: Automatic calibration
The procedure that optimizes an objective function by systematically searching the parameter space according to a fixed set of rules
Example
Monte Carlo method
Genetic Algorithms
Shuffled Complex Evolution

13. Calibration approaches: Manual VS Automatic calibration
“The debate on whether manual or automatic calibration of hydrological models is superior is likely to remain inconclusive. However, better understanding of the catchment is the key for the success of both approaches.” Masih, I.,

14. A note of caution on the use of objective functions (from Krause et al
“In general, many efficiency criteria contain a summation of the error term (difference between the simulated and the observed variable at each time step) normalized by a measure of the variability in the observations. To avoid the canceling of errors of opposite sign, the summation of the absolute or squared errors is often used for many efficiency criteria. As a result, an emphasis is placed on larger errors while smaller errors tend to be neglected. Since errors associated with high streamflow values tend to be larger than those associated with errors for lower values, calibration (both manual and automatic) attempts aimed at minimizing these types of criteria often lead to fitting the higher portions of the hydrograph (e.g., peak flows) at the expense of the lower portions (e.g., baseflow). Further, different efficiency criterion may place emphasis on different systematic and/or dynamic behavioural errors making it difficult for a hydrologist to clearly assess model performance.”

15. A note of caution on the use of objective functions
Every objective functions has strengths and weakness
What we can do to overcome these limitations
Select appropriate objective function according to the study objectives
Use more than one objective function (Multi-objective calibration)
Use more than one variable in calibration (e.g. ET and Q)
Calibrate at more than one observation point (e.g. more flow gauges within a basin than only at one station)
What else?

16. Examples of model calibration and evaluation: Masih et al
Examples of model calibration and evaluation: Masih et al., 2010-HBV model application, Iran

17. Examples of model calibration and evaluation: Masih et al
Examples of model calibration and evaluation: Masih et al., 2010-HBV model application, Iran
Calibration (Oct Sep. 1994)
R2= 0.91; NSE = 0.91; Bias= -5%
Observed (blue)
Simulate (red)

18. Some examples: Model calibration and evaluation
Example from Masih et al., 2011-SWAT model application: What do you observe by visually inspecting these graphs
Which one is good or bad?
What could be the reason?
Can we improve the fit through calibration process or this is what we can achieve at best?

19. HYPE model application in Swedon, Lindsrom et al., 2010
Figure 5 | Examples of local model evaluations from different river basins in Sweden (gray: simulation and black: observations). The model was optimized to the observations in each particular diagram. S: snow depth at Ljusnedal (R 2 = 0.92); E: evaporation at Norunda (R 2 = 0.59; Grelle 2008); G: groundwater level at Svartberget (R 2 = 0.90); F: soil frost depth at Svartberget (R 2 = 0.84, Lindstro¨m et al. 2002); W: lake water level in Lake Mo¨ ckeln (R 2 = 0.95); Q: discharge at Torsebro (R 2 = 0.94); O18: 18O content at Stubbetorp (R 2 = 0.54, Andersson & Lepisto¨ 1998); TN: total nitrogen at JRK research basin O19 (R 2 = 0.68) and TP: total phosphorous at JRK research basin AB5 (R 2 = 0.41).

20. Validation
Model should be able to reproduce field observations not used in calibration
Can be carried out as long as an independent data set has been kept aside

21. Validation Ensuring theories and assumptions are correct
Ensuring computer programming code is correct
Water balance calculation
Parameters are valid values
Compare to second data series of observed data

22. Examples model validation: Masih et al
Examples model validation: Masih et al., 2010-HBV model application, Iran. Same parameter set used for the validation as of calibration
Validation (Oct.1994-Sep. 2001)
R2= 0.81; NSE = 0.67; Bias= 20%
Observed (blue)
Simulate (red)

23. Examples model validation: Merz and Bloschl, 2004-HBV model application to 308 Austrian catchments

24. Model Evaluation: Calibration and validation
If simulations are good in both calibration and validation period, it gives an indication of the possibility of a model to be confidently used for the intended purpose
Most often the model performance during validation period is lower than calibration, but could be other way around as well.
Modeler should try to find reasons for high/low performance
Sensitivity and uncertainty analysis are also integral part of calibration and validation process, and should be properly included in a modelling study.

25. Discussion and Questions