1. Generation of Pareto Optimal Ensembles of Calibrated Parameter Sets for Climate Models Keith Dalbey, Ph.D. Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification Michael Levy, Ph.D. Sandia National Labs, Dept 1442, Numerical Analysis and Applications Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energys National Nuclear Security Administration under Contract DE-AC04-94AL85000. December 12-17, 2010

2. Outline Motivation Approach: Pareto Ensemble What Does Pareto Optimal Mean? Finding a Pareto Optimal Ensemble Results of Tuning Climate Model Summary & Future Work References Jackson et al, Error reduction and convergence in climate prediction, Journal of Climate, 2008. Eddy & Lewis, Effective generation of pareto sets using genetic programming, Proc. of ASME Design Engineering Technical Conference, 2001. Dalbey & Karystinos, Fast generation of space-filling latin hypercube sample designs, Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2010.

3. Motivation Calibrating (tuning) climate models choosing values of model parameters to predict well Is difficult because They have many inputs and outputs Diverse parameters sets can match observations similarly well Errors can compensate: 2 wrongs can make a right under historical conditions Climate change (new conditions) might expose a previously hidden mis-calibration, so… History matching is necessary but not sufficient for good predictions. The future is uncertain, but we can quantify the uncertainty (estimate statistics) for possible future climates.

4. Approach: Pareto Ensemble How can we make good statistical predictions? Use a diverse ensemble of good parameter sets to determine the range/spread of possible future climates QUESTION: Whats the definition of a good parameter set? There are multiple outputs and whats good for one output can be bad for another. (AN) ANSWER: Its Pareto optimal. A point (parameter set) is Pareto optimal if there is no other point that is as good or better than it in ALL outputs. What does the Pareto mean? Its just the name of the person who discovered it… Vilfredo Federico Damaso Pareto was an Italian engineer, sociologist, economist, and philosopher.

5. What Does Pareto Optimal Mean? 2D Pareto front schematics

6. What Does Pareto Optimal Mean? Usually, the current approx. of the true Pareto front. The Pareto front defines the zero sum game of all optimal compromises you could make. Unlike a weighted combination of objective functions, it lets you choose a specific compromise/ combination AFTER the optimization is complete. It does NOT say which compromise/combination is best, just what all the optimal choices are. It says Dont choose anything Pareto non-optimal because theres something better in all criteria.

7. Finding a Pareto Optimal Ensemble Used the Multi Objective Genetic Algorithm (MOGA) in DAKOTAs ( Design Analysis Kit for Optimization and Terascale Applications ) JEGA ( John Eddys Genetic Algorithm) sub-package GAs typically need 1000s of simulations, I could only afford 1000… Used test problem (find surface of radius=1 6D hyper- sphere in input space, 10 outputs) to tune MOGA settings and initial population ( space-filling, specifically Binning Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS ), for: Large Pareto Ensemble Mean radius close to 1 Uniform spread Small radius variance

8. Finding a Pareto Optimal Ensemble Use DAKOTAs MOGA on a test problem with 6 inputs and 10 outputs; true solution is a radius 1 hypersphere Default Monte Carlo seed PDFs of the Pareto Ensembles 1.# of points 2.Point spread 3.Mean radius 4.Standard deviation of radius 12 34

9. Finding a Pareto Optimal Ensemble Use DAKOTAs Multi Objective Genetic Algorithm on a test problem with 6 inputs and 10 outputs true solution is a radius 1 hypersphere BOSLHS seedDefault Monte Carlo seed

10. Results of Tuning Climate Model

11. Summary & Future Work Climate model parameters that match history well might not predict well (climate change might expose a previously hidden mis-calibration of parameters). Plan: Use a diverse ensemble of good (Pareto optimal) parameter sets to determine the range/spread of possible future climates. Used MOGA to find a (very large) Pareto optimal ensemble of calibrated parameter sets. Next steps: –down select Pareto optimal ensemble, and –simulate smaller ensemble out to 2100.

12. Some Good Parameter Sets