1. Overview of PEST and its Supporting Software

2. Setting the context: decision-support environmental modelling
Groundwater
Surface water
Land use
etc

3. Modelling for Decision Support
Requirements of decision-support modelling
Identify predictions of management interest
Assimilate data that can reduce the uncertainties of these predictions
expert knowledge (a stochastic quantity)
historical measurements of system state and fluxes
Quantify uncertainties associated with these predictions
Risk = cost of failure × probability of failure

4. Model Dancing Partners Model Dancing Partners
Global sensitivity analysis
Calibration-constrained
Monte Carlo
Optimisation under uncertainty
Direct predictive hypothesis testing
etc
Inversion
Requirements: hundreds (maybe thousands) of model runs

5. Model Uncertainty Quantification

6. P(k|h)  P(h|k) P(k) History-matching: Bayes Equation History-matching
Posterior
Likelihood
Prior
Which of these possibilities fits the data
What is possible based on expert knowledge
The possibilities that remain

7. Bayes Equation So what does the “calibrated model” provide?
Posterior predictive probability distribution
Prior predictive probability distribution
So what does the “calibrated model” provide?

8. The “Calibrated Model”
Predictions of minimum error variance
(if we do it properly)
a simplified, abstract parameter field
that captures all information available in a calibration dataset (which is insufficient to describe a system)
and whose relationship to reality is expressed by an unknown projection operator onto a limited dimensional space

9. The “Calibrated Model”
A bad thing
Predictions of minimum error variance
(if we do it properly)
But what is the error variance?

10. The “Calibrated Model”
A bad thing
Predictions of minimum error variance
(if we do it properly)
But what is the error variance?

11. The “Calibrated Model”
A bad thing
Predictions of minimum error variance
(if we do it properly)
But what is the error variance?

12. Modelling for Decision Support
Some notes on assessing predictive uncertainty
A model cannot say what will happen in the future; it can only say what will not happen in the future
Parameter nonuniqueness is often the dominant contributor to the uncertainty of predictions made by an environmental model (particularly a groundwater model)
Therefore, when assessing uncertainty, the parameters that cannot be estimated are as important as those that can
It is therefore almost mandatory to endow an environmental model (particularly a groundwater model) with a large number of parameters

13. What makes a good model dancing partner?

14. Model dancing partners
A non-intrusive interface with the model
The ability to parallelize model runs

15. Input files Model Output files Non-intrusive model interface
These must be text files
The model must be capable of being run using a system command.
Model
Output files
These must be text files

16. (e.g. PEST) Template files Input files Model Output files
Non-intrusive model interface
Writes model input files
Template files
Input files
These must be text files
(e.g. PEST)
The model must be capable of being run using a system command.
Model
Output files
These must be text files
Instruction files
Reads model output files

17. Non-intrusive model interface
template file 1
template file 2
template file 3
template file 4
input file 1
input file 2
input file 3
input file 4
program1.exe
program3.exe
program2.exe
program4.exe
program4.exe
output file 1
output file 2
output file 3
instruction file 1
instruction file 2
instruction file 2

18. Model run parallelization
Same computer
Office network
High performance cluster
The Cloud
TCP/IP
Inversion/uncertainty analysis/optimization engine and run manager

19. Model run parallelization
Input files
Input files
Input files
Input files
Model
Model
Model
Model
Output files
Output files
Output files
Output files
Agent #1
Agent #2
Agent #3
Agent #4
TCP/IP
Inversion/uncertainty analysis/optimization engine and run manager
Agent #5
Agent #6
Input files
Input files
Model
Model
Output files
Output files

20. PEST

21. PEST PEST modes of operation Parameter estimation Predictive analysis
Estimates parameters where inverse problem is well-posed
PEST modes of operation
Parameter estimation
Predictive analysis
Regularization
Pareto
Maximizes/minimizes the value of a prediction while maintaining the calibration objective function below a certain value
Estimates parameters where inverse problem is ill-posed
Direct predictive hypothesis-testing, or balancing measurement-to-measurement fit against regularization constraints

22. PEST Some PEST specifications
Can estimate thousands of parameters on the basis of tens of thousands of observations
Arbitrary observation weighting, including covariance matrices instead of weights
Regularization devices include:
Tikhonov
Singular value decomposition
LSQR
Sensitivities are calculated using finite differences, or can be passed from a model
Two, three and five point finite-difference stencils
Tolerance of model output granularity
Comprehensive record of inversion process
Parallelized model runs can be undertaken for any purpose

23. PEST More PEST features
The number of model runs required for parameter estimation can be reduced through
sensitivity re-use
simultaneous parameter increments
use of “super parameters” defined through singular value decomposition
construction of a pseudo-sensitivity matrix from random parameter realizations
Sophisticated handling of parameter bounds
Parameters can be tied or fixed
Strategic file transfer between manager and parallel agents
Conjunctive use of complementary simple and complex models
Primary parameters, secondary parameters and “file parameters” can be defined
A PEST run can be stopped and re-started
etc

24. (e.g. PEST) Template files PEST control file Input files Model
File types required by PEST
Writes model input files
Template files
PEST control file
Input files
These must be text files
(e.g. PEST)
The model must be capable of being run using a system command.
Model
Output files
These must be text files
Instruction files
Reads model output files

25. PEST is supplied as:
WINDOWS executables (32 bit and 64 bit)
Source code and UNIX makefiles

26. Other PEST-Suite Dancing Partners
Name of Program
What it Does
CMAES
Global optimization using the covariance matrix adaptation method
CMAES_HP
Enhanced version of CMAES (includes “file parameters” and better model run parallelization)
SCE_UA
Global optimization using shuffled complex evolution scheme
JACTEST
Tests for model output granularity
JACTEST_HP
JACTEST with superior parallelization capabilities
RSI_HP
“Realization space inversion” – efficient sampling of posterior parameter probability distributions
AGENT_HP
General agent used in model run parallelization

27. Manuals for PEST

28. PEST++ Suite Dancing Partners
Name of Program
What it Does
PESTPP-GLM
Highly parameterized, regularized inversion
PESTPP-IES
Advanced ensemble smoother
PESTPP-SEN
Global sensitivity analysis
PESTPP-OPT
Decision optimization under chance constraints
PESTPP-SWP
Model runs for any purpose
All are compatible with PEST:
read a PEST control file
use template and instruction files
parallelize model runs using manager and agent concept

29. PEST Support Utilities
(over 130)

30. PEST Utilities All of these, like PEST, are discrete programs
WINDOWS executables are provided
Source code can also be provided so that they can be compiled for use on any system

31. PEST Utilities Task categories
Construction, manipulation, editing and checking of PEST input datasets
Manipulation of PEST-calculated parameter sensitivities
Model pre- and post-processing
Linear parameter and predictive uncertainty and error analysis
Nonlinear parameter and predictive uncertainty and error analysis
Analysis of PEST results as recorded on its output files
Miscellaneous other tasks

32. PEST Utilities
Construction, manipulation, editing and checking of PEST input datasets
Build a PEST control file
Adjust weights in a PEST control file
Reconfigure weights to balance observation group objective functions
Reconfigure weights to reflect details of model-to-measurement fit achieved through history matching
Add Tikhonov regularization to a PEST control file
Insert optimized parameters into a new PEST control file
Check all or part of a PEST input dataset for integrity and consistency

33. PEST Utilities Manipulation of PEST-calculated parameter sensitivities
(These are contained in a “JCO” file)
Extract rows and columns from a JCO file
Build a JCO file from its parts
Create a JCO file for a new PEST control file from an existing JCO file
Build a rank-deficient JCO file from parameter-to-model-output covariances
Check a JCO file for consistency with a PEST control file
Build a weighted JCO file from a non-weighed JCO file

34. PEST Utilities Model pre- and postprocessing
On-the-fly computation of “secondary parameters” used by model from primary parameters estimated by PEST
On-the-fly processing of model outputs to compute “secondary outputs” which can then be matched with field measurements

35. PEST Utilities Linear parameter/predictive uncertainty/error analysis
Assess prior and posterior predictive uncertainties
Assess posterior parameter uncertainties
Compute contributions to prior/posterior predictive uncertainty by different parameter groups
Compute contributions to predictive uncertainty from measurement noise
Compute contributions to predictive uncertainty incurred by lack of information in a calibration dataset
Compute the ability of existing and not-yet-existing data to reduce the uncertainties of user-specified predictions
Calculate parameter identifiability
Explore causes of model-to-measurement misfit
Explore the effects of model defects on pre- and post-calibration model predictive bias
Calculate the number of separate pieces of information in a calibration dataset
Track flow of information from observations to parameters (“super observations” and “super parameters”)

36. Some outcomes of linear analysis
Contributions of different parameter types to the uncertainty of a prediction of interest

37. Some outcomes of linear analysis
Parameter identifiabilities:
(colour-coded according to singular values associated with solution space eigenvectors)

38. Some outcomes of linear analysis
Relative parameter uncertainty reduction
(locations of borehole head measurements are also shown)

39. PEST Utilities
Nonlinear parameter and predictive error and uncertainty analysis
Generation of random parameter sets
Sampling a linear approximation to the posterior parameter probability distribution
Null space projection of random parameter fields to reduce model-to-measurement misfit
Automatic adjustment of random parameter fields to fit measurement dataset
Pre/postprocessing for PESTPP-IES ensemble smoother and RSI_HP realization space inverter
Assistance with direct predictive hypothesis testing
Reading of multiple model output files generated through Monte Carlo analysis and tabulation of results
Data space inversion

40. PEST Utilities Analysis of PEST outcomes
Calculation of influence statistics
Advice on PEST settings
Re-computation of objective functions based on revised weighting strategies
Assessment of the integrity of finite-difference derivatives

41. Groundwater Utilities

42. PEST Groundwater Utilities
Like the previous set of utilities, all of these are discrete programs
WINDOWS executables are provided
Source code can also provided so that they can be compiled for use on any system
Documentation is extensive…..

44. Underlying principles
PEST Groundwater Utilities
Underlying principles
Benefits of using a large number of parameters
With the help of appropriate Tikhonov regularization, heterogeneity “finds itself”
The calibrated model makes predictions of minimized error variance
The chances of predictive bias introduced by parameter parsimony are reduced
Parameter superfluity allows establishment of the null space
This helps to guarantee that predictive uncertainty is not under-estimated

45. Underlying principles
PEST Groundwater Utilities
Underlying principles
Benefits of formulating a creative, multi-component objective function
Reduce parameter and predictive uncertainty
Protect the calibration process from bias induced by model defects
Enhance calibration efficiency by reducing nonlinearity between parameters and model outputs
One-sided “penalty functions” can facilitate user involvement in the inversion process

46. PEST Groundwater Utilities
Assist in PEST setup for commonly used groundwater models. These models include:
Structured-grid versions of MODFLOW
Unstructured-grid versions of MODFLOW (MODFLOW-USG, MODFLOW6)
MT3D
SEAWAT
FEFLOW
TOUGH2
HydroGeoSphere
HYDRUS
and other models

47. Pilot points parameterization
PEST Groundwater Utilities
Pilot points parameterization
Support for pilot points parameterization
Two- and three-dimensional interpolation from pilot points to model grid
Interpolation using kriging, radial basis functions and inverse power of distance
Multi-stage parameter definition, including recognition of inter-parameter dependencies
Arbitrary mathematical manipulation of pilot point and model parameter arrays
Tikhonov regularization based on “preferred parameter differences”, “preferred parameter values” and/or user specified inter-parameter relationships
Formulation of covariance matrices for regularization and uncertainty analysis based on arbitrarily complex and spatially varying variograms

48. A MODFLOW-USG Model

49. A MODFLOW-USG Model

50. A MODFLOW-USG Model

51. A MODFLOW-USG Model
Near-pit pilot points (3d)

52. A MODFLOW-USG Model
Regional pilot points (2d)

53. A MODFLOW-USG Model
Pilot points assigned to individual faults (2d)

54. A MODFLOW-USG Model
All pilot points

55. A MODFLOW-USG Model
Shaded according to log of Kh

56. A MODFLOW-USG Model
Shaded according to log of Kh

57. A MODFLOW-USG Model
Shaded according to log of Kh

58. A MODFLOW-USG Model
Shaded according to log of Kh

59. A MODFLOW-USG Model
Shaded according to log of Kh

60. A MODFLOW-USG Model
Shaded according to log of Kh

61. PEST Groundwater Utilities
Some other tasks
Some other tasks undertaken by PEST groundwater utilities
Reading of MODFLOW/MODFLOW-USG/MODFLOW6/MT3D binary head, budget and other model output files
Spatial and temporal interpolation of model results to sites and times of measurements
Tabulation of model outputs for easy input to plotting and graphing software
Re-formatting of model inputs and outputs for use by visualization, display and GIS software

62. Surface Water Utilities

63. PEST Surface Water Utilities
Like the previous set of utilities, all of these are discrete programs
WINDOWS executables are provided
Source code can also provided so that they can be compiled for use on any system
Documentation is extensive…..

65. Underlying principles
PEST Surface Water Utilities
Underlying principles
Calibration and uncertainty analysis of surface water and land use models
Benefits of a highly-parameterized approach
Parameter “regionalization relationships” can be introduced through Tikhonov regularization and simultaneous calibration over multiple watersheds
Uncertainties of predictions (particularly of high and low flow extremes) are not under-estimated because of parsimonious parameterization

66. Calibrating a multi-watershed surface water model
“Lumped element” model
(e.g. SYMHYD)

67. Calibrating a multi-watershed surface water model
“Lumped element” model
(e.g. SYMHYD)
Option 1:
Calibrate watersheds separately in order 1, 2, 3
Problem
Parameter nonuniqueness in each watershed
No respect paid for possible similarity of parameters across watersheds

68. Calibrating a multi-watershed surface water model
“Lumped element” model
(e.g. SYMHYD)
Option 2:
Calibrate watersheds together using regularization that expresses preferred parameter similarity across watersheds, but allows departures from this if necessary to fit flows
Benefits
Promotes regionalization of parameters (parameters actually “mean something”)
Reduces parameter nonuniqueness

69. Calibrating a multi-watershed surface water model
Land uses within watersheds
A more elaborate option:
Assign different models to different land uses in different watersheds
Ascribe similarity relationships to parameters of same type pertaining to same land use in different watersheds
Ascribe expert-knowledge-based ordering relationships between parameters of same type in different land uses in same watershed
Estimate all parameters for all land uses for all watersheds simultaneously subject to Tikhonov constraints that minimize departures from regularization conditions to those required to allow replication of observed flows
Benefits
Maximum use of expert knowledge in parameter assignment
Good fit with historical flows
Firm basis for post-calibration uncertainty analysis (the many parameters informed by expert knowledge respects the complexity of the system)
“Lumped element” model
(e.g. SYMHYD)

70. Underlying principles
PEST Surface Water Utilities
Underlying principles
Calibration and uncertainty analysis of surface water and land use models
The need for an innovative, multi-component objective function
Different aspects of a flow time-series that are information-rich in their own ways are given a voice in the parameter estimation process
These include flow statistics (e.g. exceedance times) that may form the basis for management-critical predictions
The inversion process can thus “tune” a model for optimality of use in a particular decision- making context

71. Where is the information?
Surface resistance to overland flow
Size of soil moisture store
Properties of medium through which interflow and/or shallow groundwater flow takes place
Properties of deep groundwater system
Evaporative and other losses

72. PEST Surface Water Utilities
TSPROC – a time series processor built for model calibration
Some specifications
Reads time series from files of many formats
Undertakes temporal interpolation from modelled to measured flows
Calculates one time series from multiple other time series using arbitrary mathematical relationships
Calculates flow volumes over arbitrary time intervals
Undertakes digital filtering of flows
Undertakes baseflow separation
Calculates statistics based on time series
Generates a PEST input dataset based on a multi-component objective function which incorporates all of the above

73. Conclusions

74. Decision-support modelling requires:
Quantification of the uncertainties of decision-critical model predictions
Assimilation of information that can reduce these uncertainties
This, in turn, requires the use of model-partner software which:
Is non-intrusive
Supports parallelization of model runs
Whose use is facilitated by utility software that assists in setup of:
Complex inversion and uncertainty analysis problems
Complementary tasks such as linear and nonlinear parameter and predictive uncertainty analysis

75. The End