Generation of Pareto Optimal Ensembles of Calibrated Parameter Sets for Climate Models Keith Dalbey, Ph.D. Sandia National Labs, Dept 1441, Optimization.

Slides:

Advertisements

Similar presentations

New Micro Genetic Algorithm for multi-user detection in WCDMA AZMI BIN AHMAD Borhanuddin Mohd Ali, Sabira Khatun, Azmi Hassan Dept of Computer and Communication.

Advertisements

© Telelogic AB [1] Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company for the United States Department of Energys.

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energys National Nuclear.

Conclusion Kenneth Moreland Sandia National Laboratories Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,

Parallel Visualization Kenneth Moreland Sandia National Laboratories Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin.

Aerospace Vehicle Ground Testing Sandia National Laboratories Trisonic Wind Tunnel (TWT) Technical Characteristics Blowdown to atmosphere M = ,

1 -Classification: Internal Uncertainty in petroleum reservoirs.

Von Karman Institute for Fluid Dynamics RTO, AVT 167, October, R.A. Van den Braembussche von Karman Institute for Fluid Dynamics Tuning of Optimization.

1 Uncertainty in rainfall-runoff simulations An introduction and review of different techniques M. Shafii, Dept. Of Hydrology, Feb

Photos placed in horizontal position with even amount of white space between photos and header Sandia National Laboratories is a multi-program laboratory.

Bridging the Gap Between Statistics and Engineering Statistical calibration of CFD simulations in Urban street canyons with Experimental data Liora Malki-Epshtein.

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation,

STRUCTURAL AND MULTIDISCIPLINARY OPTIMIZATION GROUP

Space-Filling DOEs These designs use values of variables inside range instead of at boundaries Latin hypercubes, one popular space- filling DOE uses as.

Design Rule Generation for Interconnect Matching Andrew B. Kahng and Rasit Onur Topaloglu {abk | rtopalog University of California, San Diego.

Combining Monte Carlo Estimators If I have many MC estimators, with/without various variance reduction techniques, which should I choose?

Designing Ensembles for Climate Prediction

Ensemble Emulation Feb. 28 – Mar. 4, 2011 Keith Dalbey, PhD Sandia National Labs, Dept 1441 Optimization & Uncertainty Quantification Abani K. Patra, PhD.

Use of Kalman filters in time and frequency analysis John Davis 1st May 2011.

Model calibration using. Pag. 5/3/20152 PEST program.

Variance reduction techniques. 2 Introduction Simulation models should be coded such that they are efficient. Efficiency in terms of programming ensures.

Optimal Design Laboratory | University of Michigan, Ann Arbor 2011 Design Preference Elicitation Using Efficient Global Optimization Yi Ren Panos Y. Papalambros.

Engineering Economic Analysis Canadian Edition

1 Approved for unlimited release as SAND C Verification Practices for Code Development Teams Greg Weirs Computational Shock and Multiphysics.

Department of Engineering, Control & Instrumentation Research Group 22 – Mar – 2006 Optimisation Based Clearance of Nonlinear Flight Control Laws Prathyush.

MAE 552 Heuristic Optimization Instructor: John Eddy Lecture #36 4/29/02 Multi-Objective Optimization.

Addressing Appropriate Criteria in Design: A Blend of Organization Consciousness and Awareness Kemper Lewis Associate Professor Department of Mechanical.

Design of Curves and Surfaces by Multi Objective Optimization Rony Goldenthal Michel Bercovier School of Computer Science and Engineering The Hebrew University.

Random Number Generators CISC/QCSE 810. What is random? Flip 10 coins: how many do you expect will be heads? Measure 100 people: how are their heights.

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear.

LTE Review (September 2005 – January 2006) January 17, 2006 Daniel M. Dunlavy John von Neumann Fellow Optimization and Uncertainty Estimation (1411) (8962.

01/24/05© 2005 University of Wisconsin Last Time Raytracing and PBRT Structure Radiometric quantities.

Prof. Dr. S. K. Bhattacharjee Department of Statistics University of Rajshahi.

Monte Carlo Simulation and Personal Finance Jacob Foley.

Copyright © 2010 Lumina Decision Systems, Inc. Monte Carlo Simulation Analytica User Group Modeling Uncertainty Series #3 13 May 2010 Lonnie Chrisman,

IV&V Facility PI: Katerina Goseva – Popstojanova Students: Sunil Kamavaram & Olaolu Adekunle Lane Department of Computer Science and Electrical Engineering.

Probabilistic Mechanism Analysis. Outline Uncertainty in mechanisms Why consider uncertainty Basics of uncertainty Probabilistic mechanism analysis Examples.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1 Part 4 Curve Fitting.

Physics 270 – Experimental Physics. Standard Deviation of the Mean (Standard Error) When we report the average value of n measurements, the uncertainty.

Lecture 19: Free Energies in Modern Computational Statistical Thermodynamics: WHAM and Related Methods Dr. Ronald M. Levy Statistical.

Assimilation of HF Radar Data into Coastal Wave Models NERC-funded PhD work also supervised by Clive W Anderson (University of Sheffield) Judith Wolf (Proudman.

Progress Towards Nested Space and Sub-Space Filling Latin Hypercube Sample Designs July 25 – July 28, 2011 Keith Dalbey, PhD Sandia National Labs, Dept.

Engineering Economic Analysis Canadian Edition

Hit-and-Miss (or Rejection) Monte Carlo Method:

Hit-and-Miss (or Rejection) Monte Carlo Method: a “brute-force” method based on completely random sampling Then, how do we throw the stones and count them.

1 Everyday Statistics in Monte Carlo Shielding Calculations  One Key Statistics: ERROR, and why it can’t tell the whole story  Biased Sampling vs. Random.

Center for Radiative Shock Hydrodynamics Fall 2011 Review Assessment of predictive capability Derek Bingham 1.

Mathematical Models & Optimization?

5 Descriptive Statistics Chapter 5.

EXTENSION OF LATIN HYPERCUBE SAMPLES WITH CORRELATED VARIABLES C. J. SALLABERRY, a J. C. HELTON b – S. C. HORA c aSandia National Laboratories, New Mexico.

Peter Knippertz et al. – Uncertainties of climate projections of severe European windstorms European windstorms Knippertz, Marsham, Parker, Haywood, Forster.

5-1 ANSYS, Inc. Proprietary © 2009 ANSYS, Inc. All rights reserved. May 28, 2009 Inventory # Chapter 5 Six Sigma.

Fast Generation of Nested Space-filling Latin Hypercube Sample Designs

Monte Carlo Process Risk Analysis for Water Resources Planning and Management Institute for Water Resources 2008.

Sandia is a multi-program laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear.

Multifidelity Optimization Via Pattern Search and Space Mapping Genetha Gray Computational Sciences & Mathematics Research Sandia National Labs, Livermore,

Reservoir Uncertainty Assessment Using Machine Learning Techniques Authors: Jincong He Department of Energy Resources Engineering AbstractIntroduction.

State Estimation and Kalman Filtering Zeeshan Ali Sayyed.

Options and generalisations. Outline Dimensionality Many inputs and/or many outputs GP structure Mean and variance functions Prior information Multi-output,

Reducing the risk of volcanic ash to aviation Natalie Harvey, Helen Dacre (Reading) Helen Webster, David Thomson, Mike Cooke (Met Office) Nathan Huntley.

Statistics Presentation Ch En 475 Unit Operations.

More on regression Petter Mostad More on indicator variables If an independent variable is an indicator variable, cases where it is 1 will.

Evaluating Hypotheses. Outline Empirically evaluating the accuracy of hypotheses is fundamental to machine learning – How well does this estimate its.

Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.

CIVE Engineering Mathematics 2.2 (20 credits) Statistics and Probability Lecture 6 Confidence intervals Confidence intervals for the sample mean.

A Kriging or Gaussian Process emulator has: an unadjusted mean (frequently a least squares fit: ), a correction / adjustment to the mean based on data,

L-Dominance: An Approximate-Domination Mechanism

Statistical Methods For Engineers

Results of Eddy Current Analysis

Memory-Based Learning Instance-Based Learning K-Nearest Neighbor

Presentation transcript:

Generation of Pareto Optimal Ensembles of Calibrated Parameter Sets for Climate Models Keith Dalbey, Ph.D. Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification Michael Levy, Ph.D. Sandia National Labs, Dept 1442, Numerical Analysis and Applications Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energys National Nuclear Security Administration under Contract DE-AC04-94AL December 12-17, 2010

Outline Motivation Approach: Pareto Ensemble What Does Pareto Optimal Mean? Finding a Pareto Optimal Ensemble Results of Tuning Climate Model Summary & Future Work References Jackson et al, Error reduction and convergence in climate prediction, Journal of Climate, Eddy & Lewis, Effective generation of pareto sets using genetic programming, Proc. of ASME Design Engineering Technical Conference, Dalbey & Karystinos, Fast generation of space-filling latin hypercube sample designs, Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2010.

Motivation Calibrating (tuning) climate models choosing values of model parameters to predict well Is difficult because They have many inputs and outputs Diverse parameters sets can match observations similarly well Errors can compensate: 2 wrongs can make a right under historical conditions Climate change (new conditions) might expose a previously hidden mis-calibration, so… History matching is necessary but not sufficient for good predictions. The future is uncertain, but we can quantify the uncertainty (estimate statistics) for possible future climates.

Approach: Pareto Ensemble How can we make good statistical predictions? Use a diverse ensemble of good parameter sets to determine the range/spread of possible future climates QUESTION: Whats the definition of a good parameter set? There are multiple outputs and whats good for one output can be bad for another. (AN) ANSWER: Its Pareto optimal. A point (parameter set) is Pareto optimal if there is no other point that is as good or better than it in ALL outputs. What does the Pareto mean? Its just the name of the person who discovered it… Vilfredo Federico Damaso Pareto was an Italian engineer, sociologist, economist, and philosopher.

What Does Pareto Optimal Mean? 2D Pareto front schematics

What Does Pareto Optimal Mean? Usually, the current approx. of the true Pareto front. The Pareto front defines the zero sum game of all optimal compromises you could make. Unlike a weighted combination of objective functions, it lets you choose a specific compromise/ combination AFTER the optimization is complete. It does NOT say which compromise/combination is best, just what all the optimal choices are. It says Dont choose anything Pareto non-optimal because theres something better in all criteria.

Finding a Pareto Optimal Ensemble Used the Multi Objective Genetic Algorithm (MOGA) in DAKOTAs ( Design Analysis Kit for Optimization and Terascale Applications ) JEGA ( John Eddys Genetic Algorithm) sub-package GAs typically need 1000s of simulations, I could only afford 1000… Used test problem (find surface of radius=1 6D hypersphere in input space, 10 outputs) to tune MOGA settings and initial population ( space-filling, specifically Binning Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS ), for: Large Pareto Ensemble Mean radius close to 1 Uniform spread Small radius variance

Finding a Pareto Optimal Ensemble Use DAKOTAs MOGA on a test problem with 6 inputs and 10 outputs; true solution is a radius 1 hypersphere Default Monte Carlo seed PDFs of the Pareto Ensembles 1.# of points 2.Point spread 3.Mean radius 4.Standard deviation of radius 12 34

Finding a Pareto Optimal Ensemble Use DAKOTAs Multi Objective Genetic Algorithm on a test problem with 6 inputs and 10 outputs true solution is a radius 1 hypersphere BOSLHS seedDefault Monte Carlo seed

Results of Tuning Climate Model

Summary & Future Work Climate model parameters that match history well might not predict well (climate change might expose a previously hidden mis-calibration of parameters). Plan: Use a diverse ensemble of good (Pareto optimal) parameter sets to determine the range/spread of possible future climates. Used MOGA to find a (very large) Pareto optimal ensemble of calibrated parameter sets. Next steps: –down select Pareto optimal ensemble, and –simulate smaller ensemble out to 2100.

Some Good Parameter Sets