NCAR Supercomputing ‘Data Center’ Project An NCAR-led computing ‘facility’ for the study of the Earth system

Outline A Problem An Opportunity –NSF’s Petascale Roadmap A Solution –Facility Proposal: Site and Cost –Partners The Scientific Payoff Next Steps –Schedule and Panels

NCAR Leadership in Supercomputing… One of the founding missions of NCAR was: “… to provide, or arrange for provision of facilities for the scientific community as a whole that whose initial cost and upkeep lie beyond the capability of individual universities or research groups. ” – Preliminary Plans for a National Institute for Atmospheric Research – NCAR Blue Book Note: the wording does not imply physical collocation. This mission does confer a responsibility that cannot be delegated - namely maintaining an complete integrated cyberinfrastructure (CI) system for modeling and data analysis that meets our scientific community ’ s needs.

Examples of NCAR simulation science today Global change climate ensembles Weather Research Forecast Geophysical Turbulence Fire storm front modeling Space weather More…

A problem NCAR Mesa Lab computer facility is quickly becoming obsolete Power, cooling and floor space will be inadequate beyond the next procurement Science is being restricted by focusing on capacity ahead of capability

CMOS Trends Continue …

Chips: Faster, Cheaper but Hotter

An Opportunity NSF’s Petascale Roadmap “Overarching Recommendation: Establish a Petascale Collaboratory for the Geosciences with the mission to provide leadership-class computational resources that will make it possible to address, and minimize the time to solution of, the most challenging problems facing the geosciences.”

NSF Conclusions NSF is committed to developing and implementing a strategic plan for cyberinfrastructure –Broad based plan involving the university, Federal agencies, vendors, and International partners ATM, OCE, and EAR take different approaches to the realization of CI for their discipline –Dependent on the readiness of the community Petascale facility is an integrating theme for the Geosciences community –High potential for the generation of new knowledge and paradigm for the conduct of research –Building and sustaining a petascale facility will be a significant challenge to budgets and technology –Consistent with NSF strategic vision for CI

A solution for NCAR A new computing facility (not at the Mesa Lab) Extensive investigations, working with consultants and internal needs resulted in a detailed set of options Provides for 5-20 years of computing (capacity and capability) diversity based on current and predicted future trends in CMOS technology Allows NCAR to reach beyond its current research scope

The facility needed Data Center Expansion Report from NCAR’s Computing and Information Systems Lab 20,000 (initial to 60,000) sq. ft. 4 (to 13) MW power + generators Cooling, etc. On 13 acres (20 year lifetime) Accommodates computers, staff, open space, initial and future requirements

Birds Eye View

Importance of Site Selection Limited selection of sites that meet criteria –Size (10-15 acres) –Electrical capacity (up to 24 MW) –Fiber optic route (dark fiber) Investigated –Marshall –Louisville –Longmont –Westminster New partners and options are now being sought –IBM –Colorado School of Mines (Water, Political Complications, Fiber Optics) (Electrical Capacity)

Cost Drivers Primary Drivers –Tier III Reliability Mechanical Systems Electrical Systems –Engineering Secondary Drivers –Building Size –Land Site Facility - up to $75M (one time) Operations - $15M/year? (2X) Computing increments - $15M/year (2X) Computing infrastructure - $5M/year

The Scientific Payoff…

A petascale computer will enable scientists to … Do credible regional climate modeling for decision support. Requires resolving individual mountain ranges and ocean western boundary currents. Model climate and weather in a fully coupled mode. Better understand the marine biogeochemical cycles. Requires resolving ocean mesoscale eddies. Accurately simulate the dynamical, microphysical and radiative cloud processes. Improve seismic predictions and understand the structure of the inner core as well as the fine structure of the lower mantle.

A petascale computer will enable scientists to Perform new research in solid earth and environmental engineering Assimilate thousands of earthquakes bringing the fine structure of the Earth’s mantle and inner core into focus. Study the physical basis of land surface parameterizations by modeling soils, topography and vegetation at sub-meter scales. More accurately predict the damaging effects of solar flares on satellites and power distribution systems by resolving the fine structure of the corona magnetic field. Investigate energy management applications

Science examples …

2005 Hurricane Katrina, Track Forecast 4 km, 62 h forecast Landfall on 8/29 14Z, Louisiana/Mississippi Border 12 km, 86 h forecast Observed Track Official Forecast

Mobile Radar Hurricane Katrina Reflectivity at Landfall 29 Aug Z 4 km WRF, 62 h forecast

Radar Composite Reflectivity WRF 4 km Hurricane Katrina 72 h Forecast Initialized 27 Aug Z WRF Max Reflectivity

72 h Rita Wind Forecast Initialized 19 Sep Z Movie removed

Coupled Climate System Model Movie removed

Integrated Space Weather Modeling Movie removed

Thus …

Main Points Huge scientific discoveries await geoscience modelers at 1 PFLOPS and beyond. CMOS continues to get hotter and cheaper. The most recent acquisition tracks this trend. Every center is (or will be) facing facility challenges in the race to these discoveries. This situation is NOT unique to NCAR. NCAR now has a facility plan, that if successful, uniquely positions it as a world leader in geoscience simulation. The new facility is not a crisis: it is an opportunity.

The Opportunity Understanding of fundamental physical processes in the Sun-Earth system Environmental and Energy applications not yet possible NCAR and partners will scope/define these options –Such a facility would be a computational equivalent of the Hubble Telescope for geoscience simulation.

Next Steps

The Schedule Formed NCAR project committee Forming Blue Ribbon Panel and hold workshop - early Oct Project plan development Oct-Dec Community engagement - Nov-Jan Formalize partnerships - Oct-Dec Present initial plan to UCAR Board and National Science Foundation, early October, Initiate facility - June 2006? First electrons - June March 2009?

The Blue Ribbon Panel Rick Anthes Kelvin Droegemeier Tamas Gombosi Thomas Jordan Jean-Bernard Minster John Orcutt Tim Palmer Jagadish Shukla David Yuen Plus some to be confirmed

Contacts at NCAR Tim Killeen - NCAR Director Lawrence Buja and Peter Fox are co-chairs of the NCAR project team Aaron Anderson is the computing facilities contact Jeff Reaves is the financial/ contracts contact

Concluding remarks …

Cost Breakdown Includes the backbone for 4MW but only half the electrical equipment 2MW Excludes inflation Additional data center modules