U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, ADVANCED SCIENTIFIC COMPUTING RESEARCH An Overview Michael Strayer Associate Director, Office of Science

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Department of Energy Organizational Structure

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Advanced Scientific Computing Research Advanced Scientific Computing Research Computational Science Research and Partnerships Facilities Small Business Research NERSC Oak Ridge LCF Argonne LCF ESnet HPC & Network Facilities & Testbeds ASCR Mission: Steward of DOE’s Computational Science, Applied Mathematics, Computer Science, High-Performance Computing and Networking Research for open science. Deploy and operate high performance computing user facilities at LBNL, ANL, and ORNL ASCR Vision: Best in class advancing science and technological innovation through modeling and simulation

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, ASCR High Performance Computing Resources High Performance Production Computing Facility (NERSC) –Delivers high-end capacity computing to entire DOE SC research community –Large number of projects (200 – 300) –Medium- to very-large-scale projects that occasionally need a very high capability –Annual allocations Leadership Computing Facilities –Delivers highest computational capability to national and international researchers through peer-reviewed Innovative and Novel Computational Impact on Theory and Computation program –Small number of projects (10 – 20) –Multiple year allocations

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Current Facilities NERSC –10 Teraflop IBM SP 375 RS/6000 (Seaborg) with 6080 processors, 7.2 terabytes aggregate memory –6.7 Teraflop IBM Power 5 (Bassi) with 888 processors, 3.5 terabytes aggregate memory –3.1 Teraflop LinuxNetworx Opteron cluster (Jacquard) with 712 processors, 2.1 terabytes aggregate memory LCF at Oak Ridge –119 teraflop Cray XT3/XT4 (Jaguar) with 11,708 dual core AMD Opteron processor nodes, 46 terabytes aggregate memory –18.5 Teraflop Cray X1E (Phoenix) with 1,024 multi-streaming vector processors, Argonne LCF –5.7 Teraflop IBM Blue Gene/L (BGL) with 2,048 PPC processors

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Future Facility Upgrades ALCF –100 teraflop IBM Blue Gene/P delivered by end of 2007 – teraflop upgrade to IBM Blue Gene/P in late 2008 LCF – Oak Ridge –Cray XT4 upgraded to 250 TF by end of 2007 –1 Petaflop Cray Baker system to be delivered by end of 2008 NERSC –100+ teraflop Cray XT4 in operation by October 2007

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Ensuring Hardware Productivity Scientific Discovery through Advanced Computing (SciDAC) Innovative and Novel Computational Impact on Theory and Experiment (INCITE)

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Create comprehensive, scientific computing software infrastructure to enable scientific discovery in the physical, biological, and environmental sciences at the petascale Develop new generation of data management and knowledge discovery tools for large data sets (obtained from scientific user and simulations) Scientific Discovery through Advanced Computing (SciDAC)

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, SciDAC Accomplishments SciDAC teams –created first laboratory-scale flame simulation in three dimensions to better understand combustion which provides 80% of the energy used in the U.S. –simulated techniques for re-fueling fusion reactors –developed new methods for simulating improvements in future particle accelerators SciDAC partnerships improved effectiveness of scientific applications codes between 275% to over 10,000% –Example: decreased time to solution for Agile- Boltzmann baseline run from 4 weeks to 4 days (8 angles, 12 energy groups with a spatial resolution of 100) The SciDAC data mining tool, Sapphire, awarded a prestigious 2006 R&D100 award Hydroxyl radical in a turbulent jet flame SciDAC Review and Scientific Discovery document numerous SciDAC accomplishments

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, SciDAC-2 Computational Collaborations to Drive Scientific Discovery Total- $61.5 M Statistics 31- SciDAC projects 9- Centers 4- Institutes 18- Efforts in 11 application areas Astrophysics, Climate, Biology, Fusion, Petabyte data, Materials & Chemistry, Nuclear physics, High Energy physics, QCD, Turbulence, Groundwater New performers ? About 60% of the funds !

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Institutes and Centers ― Attributes ― Institutes- University-led centers of excellence –Focus on major software issues –Employ range of collaborative research interactions. –Reach out to engage a broader community of scientists in scientific discovery through advanced computation and collaboration. –Conduct training/outreach in high performance computing topics. Centers for Enabling Technology- work directly with applications: –Develop to enable scientific simulation codes to take full advantage of tera- to peta- scale. –Ensure critical computer science and applied mathematics issues are addressed in a timely and comprehensive fashion. –Address issues associated with research software lifecycle.

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, SciDAC Outreach Center “ Build Collaborations to Drive Scientific Discovery” Innovative web and software services –Tools which make SciDAC researchers more effective at delivering their technologies (web hosting and authenticated wiki-like portals) –Services to promote an easy interface between SciDAC and ‘the outside computational world’ (web, , and phone central point of contact for SciDAC inquiries) Workshops, training sessions –Getting the right people together to forge collaborations Outreach

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Conference

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Innovative and Novel Computational Impact on Theory and Experiment- INCITE Initiated in 2004 Provides Office of Science computing resources to a small number of computationally intensive research projects of large scale, that can make high-impact scientific advances through the use of a large allocation of computer time and data storage Open to national and international researchers, including industry No requirement of DOE Office of Science funding Peer-reviewed 2004 Awards: 4.9 Million processor hours at NERSC awarded to three projects 2005 Awards: 6.5 Million processor hours at NERSC awarded to three projects

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, INCITE Allocations by Disciplines 95 Million processor hours allocate to 45 projects

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, INCITE New 2008 Call for Proposals for over 0.25 Billion processor hours of INCITE allocations should be announced in mid-May at

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Road to Exascale Simulation and Modeling at the Exascale for Energy Ecological Sustainability and Global Security (E 3 SGS) Initiative The planned petascale computer systems and the progress toward exascale systems provide an unprecedented opportunity for science; one that will make it possible to use computation not only as an critical tool along with theory and experiment in understanding the behavior of the fundamental components of nature but also for fundamental discovery and exploration of the behavior of complex systems with billions of components including those involving humans.

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Critical Challenges Energy- Ensuring global sustainability requires reliable and affordable pathways to low-carbon energy production, e.g. bio-fuels, fusion and fission, and distribution on a massive scale. Ecological Sustainability- The effort toward sustainability involves characterizing the conditions for balance in the climate system. The ability to fit energy production and industrial emissions within balanced global climate and chemical cycles is the major scientific and technical challenge for this century. Security- The internet, as well as the instrumentation and control systems for the energy infrastructure, are central to the well-being of our society.

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, Programmatic Themes Engage top scientists and engineers, computer scientists and applied mathematicians in the country to develop the science of complexity as well as new science driven computer architectures and algorithms tied to the needs of scientific computing at all scales. Correspondingly, recruit and develop the next generation of computational and mathematical scientists. Invest in pioneering large-scale science, modeling and simulation that contribute to advancing energy, ecology and global security. Develop scalable analysis algorithms, data systems and storage architectures needed to accelerate discovery from large-scale experiments and enable verification and validation of the results of the pioneering applications. Additionally, develop visualization and data management systems to manage the output of large-scale computational science runs and in new ways to integrate data analysis with modeling and simulation. Accelerate the build-out and future development of the DOE open computing facilities to realize the large-scale systems-level science required to advance the energy, ecology and global security program.

U.S. Department of Energy Office of Science Advanced Scientific Computing Research Program CASC, May 3, E 3 SGS Town Hall Meetings Three “town hall meetings” on the proposed E 3 SGS initiative: Lawrence Berkeley National Laboratory hosted the first meeting on April 17-18, Oak Ridge National Laboratory – May 17-18, and Argonne National Laboratory – May 31-June 1.