Presentation is loading. Please wait.

Presentation is loading. Please wait.

Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced.

Published bySami Järvinen Modified over 5 years ago

Similar presentations

Presentation on theme: "Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced."— Presentation transcript:

1 Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced Scientific Computing Advisory Committee Oak Ridge, TN November 3, 2009 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD Twitter: lsmarr

2 ANL * Calit2 * LBNL * NICS * ORNL * SDSC
Project StarGate ANL * Calit2 * LBNL * NICS * ORNL * SDSC

3 Credits Argonne National Laboratory
Network/Systems Linda Winkler Loren Jan Wilson Visualization Joseph Insley Eric Olsen Mark Hereld Michael Papka Lawrence Berkeley National Laboratory (ESnet) Eli Dart National Institute for Computational Sciences Nathaniel Mendoza Oak Ridge National Laboratory Susan Hicks Larry Smarr (Overall Concept) Brian Dunne (Networking) Joe Keefe (OptIPortal) Kai Doerr, Falko Kuester (CGLX) San Diego Supercomputer Center Science application Michael Norman Rick Wagner (coordinator) Network Tom Hutton ANL * Calit2 * LBNL * NICS * ORNL * SDSC

4 Exploring Cosmology With Supercomputers, Supernetworks, and Supervisualization
Intergalactic medium on 2 Glyr scale 40963 particle/cell hydrodynamic cosmology simulation NICS Kraken (XT5) 16,384 cores Output 148 TB movie output (0.25 TB/file) 80 TB diagnostic dumps (8 TB/file) Science: Norman, Harkness,Paschos SDSC Visualization: Insley, ANL; Wagner SDSC ANL * Calit2 * LBNL * NICS * ORNL * SDSC

5 Project StarGate Goals
Explore Use of OptIPortals as Petascale Supercomputer “Scalable Workstations” Exploit Dynamic 10 Gbs Circuits on ESnet Connect Hardware Resources at ORNL, ANL, SDSC Show that Data Need Not be Trapped by the Network “Event Horizon” Rick Wagner Mike Norman ANL * Calit2 * LBNL * NICS * ORNL * SDSC

6 Why Supercomputer Centers Shouldn’t Be Data Black Holes or Island Universes
Results are the Intellectual Property of the Investigator, Not the Center Where it was Computed Petascale HPC Machines Not Ideal for Analysis/Viz Doesn’t Take Advantage of Local CI Resources on Campuses (e.g., Triton) or at other National Facilities (e.g., ANL Eureka) ANL * Calit2 * LBNL * NICS * ORNL * SDSC

7 Opening Up 10Gbps Data Path ORNL/NICS to ANL to SDSC
Connectivity provided by ESnet Science Data Network End-to-End Coupling of User with DOE/NSF HPC Facilities

8 Challenge: Kraken is not on ESnet
StarGate Network & Hardware ALCF ESnet DOE Eureka 100 Dual Quad Core Xeon Servers 200 NVIDIA Quadro FX GPUs in 50 Quadro Plex S4 1U enclosures 3.2 TB RAM Science Data Network (SDN) > 10 Gb/s fiber optic network Dynamic VLANs configured using OSCARS rendering SDSC Challenge: Kraken is not on ESnet NICS visualization simulation NSF TeraGrid Kraken Cray XT5 8,256 Compute Nodes 99,072 Compute Cores 129 TB RAM Calit2/SDSC OptIPortal1 20 30” (2560 x 1600 pixel) LCD panels 10 NVIDIA Quadro FX 4600 graphics cards > 80 gigapixels 10 Gb/s network throughout ANL * Calit2 * LBNL * NICS * ORNL * SDSC

9 StarGate Streaming Rendering
ESnet 3 A media bridge at the border provides secure access to the parallel rendering streams. ALCF Internal gs1.intrepid.alcf.anl.gov SDSC ALCF Updated instructions are sent back to the renderer to change views, or load a different dataset. 5 2 The full image is broken into subsets (tiles). The tiles are continuously encoded as a separate movies. flPy, a parallel (MPI) tiled image/movie viewer composites the individual movies, and synchronizes the movie playback across the OptIPortal rendering nodes. 4 Simulation volume is rendered using vl3 , a parallel (MPI) volume renderer utilizing Eureka’s GPUs. The rendering changes views steadily to highlight 3D structure. 1 ANL * Calit2 * LBNL * NICS * ORNL * SDSC

10 Test animation of 1/64 of the data volume (10243 region)
ANL * Calit2 * LBNL * NICS * ORNL * SDSC

11 Data Moved ORNL to ANL data transfer nodes Additionally
577 time steps ~148TB Peak bandwidth ~2.4Gb/s Disk to disk GridFTP, Multiple Simultaneous Transfers, Each with Multiple TCP Connects Average Aggregate Bandwidth <800mb/s, Using Multiple Transfers Additionally Pre-Transfer: Data was Stored in ORNL HPSS, Had to be Staged to Disk on Data Transfer Nodes One Moved to HPSS Partition, Cant Move Data Back Post-Transfer: Each Time Step was a Tar File, Had to Untar Moving Forward, will Need Direct High-Bandwidth Path from Kraken (NICS) to Eureka (ALCF) ANL * Calit2 * LBNL * NICS * ORNL * SDSC

12 ANL Eureka Graphics Cluster
NSF TeraGrid Review January 10, 2006 ANL Eureka Graphics Cluster Data Analytics and Visualization Cluster at ALCF (2) Head Nodes, (100) Compute Nodes (2) Nvidia Quadro FX5600 Graphics Cards (2) XEON E GHz Quad Core Processors 32 GB RAM: (8) 4 Rank, 4GB DIMMS (1) Myricom 10G CX4 NIC (2) 250GB Local Disks; (1) System, (1) Minimal Scratch 32 GFlops per Server Eureka – the visualization cluster at ALCF Each node has 2 graphics cards 8 processors 32 GB RAM fast interconnect local disk Server FLOPS = 2.0 GHz * 8 cores * 2 flop per clock = 32 GFLOPS ANL * Calit2 * LBNL * NICS * ORNL * SDSC Charlie Catlett

13 Visualization Pipeline
vl3 – Hardware Accelerated Volume Rendering Library 40963 Volume on 65 Nodes of Eureka Enzo Reader can Load from Native HDF5 Format Uniform Grid and AMR, Resampled to Uniform grid Locally Run Interactively on Subset of Data On a Local Workstation, 5123 Subvolume Batch for Generating Animations on Eureka Working Toward Remote Display and Control ANL * Calit2 * LBNL * NICS * ORNL * SDSC

14 vl3 Rendering Performance on Eureka
NSF TeraGrid Review January 10, 2006 vl3 Rendering Performance on Eureka Image Size: 4096x4096 Number of Samples: 4096 Data Size Number of Processors/ Graphics Cards Load Time Render/Composite Time 20483 17 2min 27sec 9.22 sec 40963 129 5min 10sec 4.51 sec 64003 (AMR) 4min 17sec 13.42sec One of its strengths is its speed, and ability to handle large data sets. Number of procs = power of 2 to do rendering + 1 for compositing 2 graphics cards per node, so half as many nodes as listed here Data i/o is clearly the bottleneck Doing an animation of a single time step, data is only loaded once, can be pretty quick Note Data I/O Bottleneck ANL * Calit2 * LBNL * NICS * ORNL * SDSC Charlie Catlett

15 Next Experiments SC09 - Stream a 4Kx2K Movie From ANL Storage Device to OptIPortable on Show Floor Mike Norman is a 2009 INCITE investigator 6 M SU on Jaguar Supersonic MHD Turbulence Simulations for Star Formation Use Similar Data Path for This to Show Replicability Can DOE Make This New Mode Available to Other Users? ANL * Calit2 * LBNL * NICS * ORNL * SDSC

Download ppt "Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced."

Similar presentations

High Performance Cyberinfrastructure Discovery Tools for Data Intensive Research Larry Smarr Prof. Computer Science and Engineering Director, Calit2 (UC.

High Performance Cyberinfrastructure Discovery Tools for Data Intensive Research Larry Smarr Prof. Computer Science and Engineering Director, Calit2 (UC.
University of Chicago Department of Energy The Parallel and Grid I/O Perspective MPI, MPI-IO, NetCDF, and HDF5 are in common use Multi TB datasets also.

University of Chicago Department of Energy The Parallel and Grid I/O Perspective MPI, MPI-IO, NetCDF, and HDF5 are in common use Multi TB datasets also.
Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.

Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.
Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced Scientific.

Project StarGate An End-to-End 10Gbps HPC to User Cyberinfrastructure ANL * Calit2 * LBNL * NICS * ORNL * SDSC Report to the Dept. of Energy Advanced Scientific.
SAN DIEGO SUPERCOMPUTER CENTER Niches, Long Tails, and Condos Effectively Supporting Modest-Scale HPC Users 21st High Performance Computing Symposia (HPC'13)

SAN DIEGO SUPERCOMPUTER CENTER Niches, Long Tails, and Condos Effectively Supporting Modest-Scale HPC Users 21st High Performance Computing Symposia (HPC'13)
Katie Antypas NERSC User Services Lawrence Berkeley National Lab NUG Meeting 1 February 2012 Best Practices for Reading and Writing Data on HPC Systems.

Katie Antypas NERSC User Services Lawrence Berkeley National Lab NUG Meeting 1 February 2012 Best Practices for Reading and Writing Data on HPC Systems.
Cyberinfrastructure for Scalable and High Performance Geospatial Computation Xuan Shi Graduate assistants supported by the CyberGIS grant Fei Ye (2011)

Cyberinfrastructure for Scalable and High Performance Geospatial Computation Xuan Shi Graduate assistants supported by the CyberGIS grant Fei Ye (2011)
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO IEEE Symposium of Massive Storage Systems, May 3-5, 2010 Data-Intensive Solutions.

SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO IEEE Symposium of Massive Storage Systems, May 3-5, 2010 Data-Intensive Solutions.
“An Integrated West Coast Science DMZ for Data-Intensive Research” Panel CENIC Annual Conference University of California, Irvine Irvine, CA March 9, 2015.

“An Integrated West Coast Science DMZ for Data-Intensive Research” Panel CENIC Annual Conference University of California, Irvine Irvine, CA March 9, 2015.
Advanced Scientific Visualization Paul Navrátil 28 May 2009.

Advanced Scientific Visualization Paul Navrátil 28 May 2009.
HIPerSpace The Highly Interactive Parallelized Display Space.

HIPerSpace The Highly Interactive Parallelized Display Space.
Simo Niskala Teemu Pasanen

Simo Niskala Teemu Pasanen
Gordon: Using Flash Memory to Build Fast, Power-efficient Clusters for Data-intensive Applications A. Caulfield, L. Grupp, S. Swanson, UCSD, ASPLOS’09.

Gordon: Using Flash Memory to Build Fast, Power-efficient Clusters for Data-intensive Applications A. Caulfield, L. Grupp, S. Swanson, UCSD, ASPLOS’09.
Real Parallel Computers. Modular data centers Background Information Recent trends in the marketplace of high performance computing Strohmaier, Dongarra,

Real Parallel Computers. Modular data centers Background Information Recent trends in the marketplace of high performance computing Strohmaier, Dongarra,
“How LambdaGrids are Transforming Science Keynote iGrid2005 La Jolla, CA September 29, 2005 Dr. Larry Smarr Director, California Institute.

“How LambdaGrids are Transforming Science" Keynote iGrid2005 La Jolla, CA September 29, 2005 Dr. Larry Smarr Director, California Institute.
Science and Cyberinfrastructure in the Data-Dominated Era Symposium #1610, How Computational Science Is Tackling the Grand Challenges Facing Science and.

Science and Cyberinfrastructure in the Data-Dominated Era Symposium #1610, How Computational Science Is Tackling the Grand Challenges Facing Science and.
TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.

TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.
Motivation “Every three minutes a woman is diagnosed with Breast cancer” (American Cancer Society, “Detailed Guide: Breast Cancer,” 2006) Explore the use.

Motivation “Every three minutes a woman is diagnosed with Breast cancer” (American Cancer Society, “Detailed Guide: Breast Cancer,” 2006) Explore the use.
Visualization Linda Fellingham, Ph. D Manager, Visualization and Graphics Sun Microsystems Shared Visualization 1.1 Software Scalable Visualization 1.1.

Visualization Linda Fellingham, Ph. D Manager, Visualization and Graphics Sun Microsystems Shared Visualization 1.1 Software Scalable Visualization 1.1.
Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented at : Cyberinfrastructure Internship Experiences for Graduate Students Spring.

Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented at : Cyberinfrastructure Internship Experiences for Graduate Students Spring.

Similar presentations

Ads by Google