1. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY 1 Global (and Regional) Climate Modeling Overview What is the climate system? How is it modeled? What are the computational issues? John Drake Computational Climate Dynamics Group Computer Science and Mathematics Division Oak Ridge National Laboratory For Fall Creek Falls Workshop, October 27, 2003

2. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Why is DOE Interested in Climate?

3. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY

4. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Climate Change Doesn’t Just Happen A Key figure

5. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY PCM Ensembles of 21 st Century Business as Usual Results show that the global warming trend since the late 1970’s is likely to continue through much of the twenty-first century. Model includes a full range of greenhouse gas changes together with sulfate aerosol effects and no artificial flux adjustments. The pattern of surface warming from 1961-90 to 2070-99 under BAU scenario shows that warming ranges between 1 and 2 C over oceans and is above 2 C over many land areas, especially in northern high latitudes during winter where the warming is above 5 C. The meridional overturning in the North Atlantic is reduced by 20% from 1961- 90 to 2070-99… reduced local vertical mixing causes cooling over the mid- latitude North Atlantic. Ensemble-averaged precipitation in BAU shows a 20%-40% increase at high latitudes during winter and a 10%-30% decrease over subtropical dry areas.

6. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Simulation of Future Climates Computing for this simulation was done at DOE's National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory, NCAR and Oak Ridge National Laboratory Center for Computational Sciences (CCS). Credits Animation and data management: Michael Wehner/Lawrence Berkeley National Laboratory Parallel Climate Model: Warren Washington, Jerry Meehl, Julie Arblaster, Tom Bettge and Gary Strand/National Center for Atmospheric Research Visualization software Dean Williams/Lawrence Livermore National Laboratory

7. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY PCM Business as Usual and Stabilization Scenarios The ensemble-mean temperatures under the BAU and STA550 scenarios start to diverge in 2040 but become significantly different only after the mid 2060’s. Stabilized atmospheric CO2 at 550ppm by 2150 will only slow down the warming moderately during the 21 st century but could be large (1.5 C globally and 12 C in DJF at northern high-latitudes) by the later part of the 22 nd century. Dai, Meehl, Washington, Wigley, Arblaster, Ensemble Simulation of Twenty-First Century Climate Changes, BAMS Vol 82, No. 11, November 2001 Dai, Wigley, Meehl and Washington, Effects of Stabilizing Atmospheric CO2 on Global Climate in the Next Two Centuries, GRL Vol. 28, No. 23, December 2001

8. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY

9. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Why a Community Model? NCAR adopted to encourage climate research in universities using state of the art models. –1980 New components of a coupled system. - 1990 Individual projects unable to adapt to parallel architectures -2000 Pool resources and complete mission critical simulations for NSF, DOE and NASA CCSM has become the “national model” Annual Breckenridge CCSM Workshop grown to over 270 researchers

10. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY DOE SciDAC Workshop on Porting CCSM to the CRAY X1 Goals  Identify individuals and organizations engaged in porting one or more of the CCSM component models  Report progress and problems in current CCSM vectorization activities.  Identify gaps or issues in the current efforts.  Establish lines of communication between the different efforts and NCAR software engineers to encourage sharing of results and code.  Begin defining requirements and procedures for the adoption ofvector-friendly code in future released versions of CCSM. Represented: NCAR, NASA-Goddard, ORNL, LANL, LBNL, Cray, NEC, Fujitsu, CRIEPI Held Feb. 2003 in Boulder, also June in Breckenridge 2003

11. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Coupler Architecture Tony Craig, Rob Jacob, Brian Kaufman, Jay Larson, E. OngIssues: sequencingsequencing frequencyfrequency distributiondistribution parallelismparallelism single or multiplesingle or multiple executables executables stand alone executionstand alone execution Version 1.0 Released November 2002 MPH3 (multi-processor handshaking) library for coupling component models CPL6 -- Implemented, Tested, Deployed

12. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Evolution of Performance of the Community Atmospheric Model Pat Worley, John Drake Hybrid MPI/OpenMP programming paradigm Cache friendly chunks, load balance, improved algorithms

13. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Land Surface Model and River Transport Model Forrest Hoffman, Mariana Verenstein, Marcia Branstetter Community Land Model  SciDAC software engineering is focused on the interface and reduction of gather/scatters; communications bottleneck removed  Rewrite for vectorization and CLM2.2 now complete  RTM is currently single processor -- designing parallel implementation and data structures  Analysis of runoff in CCSM control simulation. Effect on July ocean salinity.

14. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY POP Ocean Model P. Jones, J. Dukowicz, J. Baumgardner, W. Lipscomb Software Engineering for POP and CICE  Design and implementation for the new ocean model (HYPOP) and CICE in progress Ocean Model Performance  POP2: new design involves a decomposition of the computational domain into blocks that can be sized to fit into cache  On 1/10 degree, SGI (2x), IBM (1.25x), long vector gets 50% peak on Fujitsu HYPOP Model Development  Treat purely Lagrangian dynamics of constant-mass layers as they inflate and deflate in regions intersecting bottom topography  Pressure gradient is split into a 'baroclinic' part that vanishes and a 'barotropic' part that does not vanish when the density is uniform  Comparison of surface height in Lagrangian and Eulerian vertical after 400 baroclinic steps oldnew Total115s55s Baroclinic93s38s Barotropic9s7s

15. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Sea Ice Model J. Schramm, P. Jones, W. Lipscomb Incremental Remapping for Sea Ice and Ocean Transport  Incremental remapping scheme that proved to be three times faster than MPDATA, total model speedup of about 30% --added to CCSM/CSIM  Cache and vector optimizations  CICE3.0 restructered for vector Community Sea Ice Model Sensitivity analysis and parameter tuning test of the CICE code  Automatic Differentiation (AD)-generated derivative code  Major modeling parameters that control the sea ice thickness computation were the ice-albedo constants, densities and emissivities of ice and snow, and salinity constant  Parameter tuning experiment with gradient information

16. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Atmospheric Chemistry P. Cameron-Smith, J. Taylor, D. Erickson, J.F. Lamarque, S. Walters, D. Rotman Gas-phase chemistry with emissions, deposition, transport and photo-chemical reactions for 89 species. Experiments performed with 4x5 degree Fvcore – ozone concentration at 800hPa for selected stations (ppmv) Mechanism development with IMPACT  A) Small mechanism (TS4), using the ozone field it generates for photolysis rates.  B) Small mechanism (TS4), using an ozone climatology for photolysis rates.  C) Full mechanism (TS2), using the ozone field it generates for photolysis rates. Zonal mean Ozone, Ratio A/C Zonal mean Ozone, Ratio B/C

17. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Ocean Biogeochemistry S. Elliot, S. Chu, M. Maltrud Iron Enrichment in the Parallel Ocean Program Surface chlorophyll distributions in POP for 1996 La Niña and 1997 El Niño

18. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Global DMS Flux from the Ocean using POP D. Erickson, J. Hernandez, M. Maltrud, S. Chu The global flux of DMS from the ocean to the atmosphere is shown as an annual mean. The globally integrated flux of DMS from the ocean to the atmosphere is 23.8 Tg S yr-1.

19. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Toward Regional Impacts Snow pack in Western U.S. reduced by greater than 50% by mid-century Higher likelihood of wintertime flooding along Cascades and Sierras Increasing model resolution Cluster analysis of PCM results

20. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Resolution and Precipitation Phil Duffy CCM3 extreme precipitation events depend on model resolution. Here we are using as a measure of extreme precipitation events the 99th percentile daily precipitation amount. Increasing resolution helps the CCM3 reproduce this measure of extreme daily precipitation events. (DJF) precipitation in the California region in 5 simulations, plus observations. The 5 simulations are: CCM3 at T42 (300 km), CCM3 at T85 (150 km), CCM3 at T170 (75 km), CCM3 at T239 (50 km), and CAM2 with FV dycore at 0.4 x 0.5 deg.

21. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Subgrid Orography Scheme Steve Ghan, Tim Shippert Reproduces orographic signature without increasing dynamic resolution Realisitic precipitation, snowcover, runoff Month of March simulated with CCSM

22. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Why Climate Prediction is Compute Limited Long time integrations:  Historical validation 1870-2000  Future scenarios 2000-2200 Comprehensive, coupled processes  Models still under development  Nonlinear feedbacks and sensitivities Multi-scale interactions Need for ensemble forecasts Decision support scenarios

23. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Leadership Class Computing Will enable  Additional atmospheric chemistry  Tropospheric  Stratospheric  Interactive land and biogeochemistry  Comprehensive carbon cycle models  Increased resolution  Atm 30 km  Ocn 1/10 degree  Lnd 1 km  Better throughput for coupled models

24. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Effect of Earth Simulator on Hardware and Software Issues Challenges assumptions  Capability computing versus capacity computing  “software is the issue”  Any code can be made to run fast on any machine. If not, change the algorithm.  Special purpose processors and vector supercomputers have run out of steam  Price - performance ratio  Mass market business model. Assertions –Vector versus cache is not the issue –Effective bandwidth and latency of memory subsystem and interconnect are key –Performance portability among platforms is possible –High percentage of peak indicates a balanced system

25. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Modeling Resource Requirements 2003 Global Coupled Models Current Model years/day 8 Storage (TB/century) 1 At current scientific complexity a century requires 12.5 days to simulate. Single researcher transfers 80Gb/day. Requires 30TB storage for year.

26. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY How Many Terabytes of Output? PCMDI Archive contains >3600 years of coupled simulation IPCC simulations beginning

27. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Infrastructure Challenges Integration with existing tools Extension of existing tools for multiple component climate system models Speed of transfers Parallelism: I/O(NetCDF) and analysis Metadata and Grid databases Analysis of ensembles Advanced statistical methods for analysis of coupled systems. CCC(2100) Current (VEMAP) UKMO Hadley(2100)

28. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Teams Using Multiple Centers DOE Supports Analysis and Archiving of Climate Data at PCMDI. NASA DAO PCMDINCAR GFDL LANL ORNL-CCS LANL NERSC NCDC Core Universities

29. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY

30. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY The End

31. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Goals of the Consortium Performance portability for the CCSM Open software design process Layered software architecture to insulate modeling Readiness for global to regional climate change simulations High fidelity ocean and ice models Extension of atmospheric chemistry capability Development of biogeochemical, terrestrial and hydrological aspects of the CCSM Towards comprehensive coupled climate simulations for study of decadal to century climate change.

32. O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Tools