1. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program Walter M. Polansky walt.polansky@science.doe.gov Office of Advanced Scientific Computing Research - Status - Petascale Simulations SciDAC U.S. Department of Energy’s Office of Science

2. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 2 American Competitiveness Initiative President Bush in the President’s State of the Union Address on January 31, 2006 “I propose to double the federal commitment to the most critical basic research program in the physical sciences over the next ten years. This funding will support the work of America’s most creative minds as they explore promising areas such as nanotechnology, supercomputing, and alternative energy sources.” Secretary Bodman commenting on President Bush’s statement “Developing revolutionary, science-driven technology is at the heart of the Department of Energy’s mission. To ensure that America remains at the forefront in an increasingly competitive world, our Department is pursuing transformational new technologies in the cutting-edge scientific fields of the 21st century—areas like nanotechnology, material science, biotechnology, and high-speed computing.”

3. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 3 Office of Science: Mission “The Office of Science plays a critical role in ensuring America’s scientific leadership and economic dynamism…” The mission of the Science program is to deliver the discoveries and scientific tools that transform our understanding of energy and matter and advance the national, economic, and energy security of the United States. Secretary Samuel Bodman, House Committee on Science Hearing, February 15, 2006 In support of it’s mission, the Science program has responsibilities in three main areas: selection and management of research; operation of world-class, state-of- the-art scientific facilities; and design and construction of new facilities. “Investment in these facilities is much more than bricks and mortar: it is an investment in discovery, and in the future of our nation.”

4. Facilities for the Future of Science: a 20-Year Outlook U.S. Department of Energy Office of Science

5. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program Office of Science -- FY2007 Budget Request- $4.1B -- FY2006 Appropriation- $3.6B

6. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 6 ASCR Vision First in Computational Science “Best in class in advancing science and technology through modeling and simulation” Facilities Enabling Technologies Computational Partnerships

7. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 7 ASCR Research Strategy Applied Mathematics and Computer Science deliver the operating systems, programming models, software tools, and mathematical algorithms and libraries needed for scientists to make effective use of petascale computing. Scientific Discovery through Advanced Computing (SciDAC) program, will strengthen activities at the software centers initiated in FY 2006 for petascale computing. SciDAC will foster additional research investments in Applied Mathematics and Computer Science to accelerate efforts in modeling and simulation on the petascale computing facilities.

8. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 8 ASCR Facilities Oak Ridge Leadership Computing Facility Argonne Leadership Computing Facility NERSC at Lawrence Berkeley National Lab ESnet at Lawrence Berkeley National Lab

9. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 9 FY 2006 ASCR Budget Distribution

10. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 10 FY 2007 ACSR Budget Status $235M $319M Dollars in Millions

11. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 11 FY 2007 - House FY 2007 House Energy and Water Development Appropriations Bill passed both the Appropriations Committee and the House Report language: “ The Committee commends the Office of Science and the Office of Advanced Scientific Computing Research for their efforts to provide cutting-edge capabilities to meet current scientific computational needs, and at the same time to extend the boundaries of that cutting edge into the next generation of high-performance scientific computers and supporting software.”

12. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 12 FY 2007 - Senate FY 2007 Senate Energy and Water Development Appropriations Bill passed the Appropriations Committee Report language “ … The Committee is concerned with the relationship between the Office of Science and the NNSA. As an example, the ASCR strategic plan discusses the need to work with other Federal agencies including several defense agencies, but only discusses in general terms three areas of research where NNSA and the Office of Science cooperated. In the area of basic research, the strategic plan states that it is an area that is `not important enough to justify ASCI investment at this time.' The Committee is also aware that the Office of Science has budgeted $13,000,000 for the DARPA to support a petaflop computer deployment by 2010. The Committee believes this funding would be better spent within the Department to support a petaflop initiative. The Department is directed to divide the funds equally between the Office of Science and the NNSA Advanced Simulation and Computing activities to support development of component architecture for high-performance software and storage”

13. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 13 Facilities Update NERSC – delivery of NERSC-5 in FY 2007 LCF at ORNL – –Upgrade Cray XT3 upgrade Now: 25Teraflop  50Teraflop By the end of 2007: 50Teraflop  250Teraflop –Acquire 1 Petascale Cray Baker system by end of 2008 LCF at Argonne – –Acquire 100 Teraflop IBM Blue Gene/P in FY2007 –Upgrade to 250-500 Teraflop IBM Blue Gene/P in 2008 –On path to a Petascale IBM Blue Gene /Q by end of the decade

14. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 14 Innovative and Novel Computational Impact on Theory and Experiment- INCITE Initiated in 2004 Provides Office of Science computing resources to a small number of computationally intensive research projects of large scale, that can make high-impact scientific advances through the use of a large allocation of computer time and data storage Open to national and international researchers, including industry No requirement of DOE Office of Science funding Peer-reviewed 2004 Awards: 4.9 Million processor hours at NERSC awarded to three projects 2005 Awards: 6.5 Million processor hours at NERSC awarded to three projects

15. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 15 INCITE 2006 Expanded to include SC high end computing resources at PNNL, ORNL and ANL in addition to LBNL and multiple year requests. Received 43 proposals requesting over 95 million processor hours. –60% from Universities –40% had funding from other federal research agencies 15 Awards for over 18.2 million processor hours Wired Magazine, August, 2006, pg. 42

16. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 16 INCITE 2007 Expanded in 2007 to include 80% of resources at Leadership Computing Facilities in addition to 10% of NERSC and 5% of PNNL Call issued July 27, 2006 Proposals due September 15, 2006 Award announcements in mid-December http://hpc.science.doe.gov

17. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 17 Scientific Discovery through Advanced Computing Structure --Scientific Application Areas - Breakthrough scientific advances through computer modeling and simulation that are impossible using theoretical and experimental studies or improve experimental science. -- Centers for Enabling Technology (CETs)- provide the essential computing and communications infrastructure to support SciDAC applications -- Institutes- University-led centers of excellence; address specific software issues; Engage a broader community of scientists in modeling and simulations. www.SciDAC.gov

18. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 18 Computing/ Networking Applications Scientific Discovery Leadership Computing- ANL 100 TF Leadership Computing- ORNL 250 TF Production Computing- NERSC 100-150 TF ESnet On path toward Dual rings 40Gbps/ 10 Gbps fault tolerant Path to Petascale - Centers for Enabling Technology - Scientific Applications Partnerships- Institutes (University-lead) - Accelerator science and simulation- Astrophysics - Climate modeling and simulation- Computational Biology - Fusion science- High-energy physics - Petabyte high-energy/nuclear physics- Materials science and chemistry - Nuclear physics- QCD - Radiation transport- Turbulence - Groundwater reactive transport modeling and simulation SciDAC-2 INCITE NERSC Allocation

19. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 19 SciDAC-2 Institutes… … University-led centers of excellence that focus on major software issues through a range of collaborative research interactions. –Develop, test, maintain, and support optimal algorithms, programming environments, systems software and tools, and applications software. –Focus on a single general method or technique. –Focal point for bringing together a critical mass of experts from multiple disciplines to focus on key problems in a particular area of enabling technologies. –Forge relationships between experts in software development, scientific application domains, high performance computing, and industrial partners. –Reach out to engage a broader community of scientists in the activities of scientific discovery through advanced computation and collaboration. –Incorporate training and outreach in high performance computing topics, including for graduate students and postdocs.

20. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 20 Centers for Enabling Technologies… (CETs) … provide the essential computing and communications infrastructure to support SciDAC applications; multi-discipline approach with activities in: –Algorithms, methods, and libraries. –Program development environments and tools -- terascale and petascale program development and tools provide maximum ease-of-use to scientific end users. –Systems software that provides system stability and functionality needed by users for tera- to peta- scale simulations. –Visualization and data management systems. CETs work directly with applications on: –Development and application of computing systems software that allows scientific simulation codes to take full advantage of the extraordinary capabilities of terascale and petascale computers. –Ensuring that the most critical computer science and applied mathematics issues are addressed in a timely and comprehensive fashion. –Addressing all aspects of the successful research software lifecycle including transition of a research code into a robust production code and long term software evolution and maintenance and end user support.

21. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 21 SciDAC Accomplishments To better understand combustion which provides 80% of the energy used in the U.S., SciDAC teams created first laboratory-scale flame simulation in three dimensions Magnetic fusion scientists and applied mathematicians simulated techniques for re-fueling fusion reactors Teams developed new methods for simulating improvements in future particle accelerators Partnerships improved effectiveness of scientific applications codes between 275% to over 10,000% Hydroxyl radical in a turbulent jet flame

22. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 22 The Power and Evolution of Partnerships Today and Tomorrow ASCR HEP NE EM NNSA BES BER FES NP Industry Methane Flames Diesel Combustion Ultrafast Science GCM Performance Clouds Simulation of Microbes Plasma Core Full Simulation of Burning Plasma Weak Matrix Elements Performance of Supernova codes Accelerator Design Increased Accuracy Elements beyond Fe LCLS System Design Neutron Transport Advanced Fuel Cycles Groundwater Transport Libraries & Algorithms Science Based Materials Improvements in Mesh Generation Virtual Prototypes

23. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 23 Road to Petascale Computing: Combustion A 50% increase in efficiency of automobiles could save 21% of U.S. oil consumption for transportation Direct Numerical Simulation of Turbulent Combustion, a collaborative effort between Sandia National Laboratories, U. of Wisconsin, U. of Michigan, U. of Maryland, U.C. Davis, and Princeton U., has been focusing on advancing science for improved fuel efficiency of gas turbines and technologies for transportation. Petascale computing will enable fundamental understanding of turbulence- chemistry interactions, e.g. multi-stage auto-ignition and flame stabilization in compression ignition engine environments, in regimes relevant to real devices. This large turbulent CO/H2 jet flame was run on the CrayX1E at ORNL LCF. (Credit for the image is to Kwan-Liu Ma, Hongfeng Yu, and Hiroshi Akiba of UC Davis.)

24. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 24 Volume rendering of scalar dissipation rates in a temporal plane jet flame. High values are shown in red, medium in yellow, and lower values in blue. (Credit: Jackie Chen/SNL) Turbulence

25. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 25 Center for Simulation of Wave Interaction with Magneto- hydrodynamics, a collaboration of PPPL, ORNL, MIT, Columbia U., Indiana U., Wisconsin U., etc., is a project for very large plasma wave simulation for fusion-based research. Using 4,096 processors of Cray XT3 at LCF (ORNL), it is the largest, most-detailed simulation ever done of plasma control waves in a tokamak, the reactor that will eventually form the core of the multinational ITER reactor. Road to Petascale Computing: Fusion

26. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 26 A multi-institution project for large- scale simulation of Neutrino-driven Supernova moves forward to discover a new instability that is important to core collapse supernovae – the supernova shock wave instability (SASI) and a plausible mechanism for generating the initial neutron star spin required to give birth to pulsars. Road to Petascale Computing: Astrophysics

27. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 27 Researchers at Argonne National Laboratory have recently developed a Portable, Extensive, Toolkit for Scientific (PETSc) numerical library used in dozen of scientific applications world-wide. PETSc encapsulates the complexity of underlying parallel algorithms and presents them in terms of numerical abstractions familiar to scientists. It has been used to extend fully compressible flow codes, most notably the pressure-oriented implicit continuous Eulerian (PCICE) scheme developed at Idaho National Engineering Laboratory. Road to Petascale Computing: Modeling for Reactor Design

28. U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC- September 6, 2006 28 Researchers at Lawrence Livermore National Laboratory have recently developed a suite of fast and scalable adaptive algorithms for modeling shocks and detonations interacting with movable rigid bodies in a fluid. Accurate 2D simulations tax existing computer resources; extending to well-resolved 3D simulations will require peta-scale resources. This new algorithm combines efficient local representations of the embedded bodies with moving geometry and adaptive mesh refinement. Road to Petascale Computing: Modeling the Interaction b/w Fluids & Embedded Solid Particles Shock hitting several movable rigid bodies using a Godunov method with adaptive mesh refinement; density and refinement grids are shown The same fluid solver can model detonations using an ignition and growth model and JWL equation of state. This mock Schlieren shows a detonation interacting with solid bodies. (Credit for the images is to Bill Henshaw (PI) and Kyle Chand, LLNL)