We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byShayna Brushwood
Modified about 1 year ago
PSC Blacklight, a Large Hardware-Coherent Shared Memory Resource In TeraGrid Production Since 1/18/2011
2 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Why Shared Memory? Enable memory-intensive computation Enable data exploration statisticsstatistics machine learning vizviz graph- based informatics Increase users’ productivity productivity algorithm expression interactivityinteractivity rapid prototyping ISV apps high- productivity languages …… Change the way we look at data Change the way we look at data Boost scientific output Broaden participation Boost scientific output Broaden participation
3 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011
4 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 PSC’s Blacklight (SGI Altix ® UV 1000) Programmability + Hardware Acceleration → Productivity 2×16 TB of cache-coherent shared memory –hardware coherency unit: 1 cache line (64B) –16 TB exploits the processor’s full 44-bit physical address space –ideal for fine-grained shared memory applications, e.g. graph algorithms, sparse matrices 32 TB addressable with PGAS languages, MPI, and hybrid approaches –low latency, high injection rate supports one-sided messaging –also ideal for fine-grained shared memory applications NUMAlink ® 5 interconnect –fat tree topology spanning full UV system; low latency, high bisection bandwidth –transparent hardware support for cache-coherent shared memory, message pipelining and transmission, collectives, barriers, and optimization of fine-grained, one-sided communications –hardware acceleration for PGAS, MPI, gather/scatter, remote atomic memory operations, etc. Intel Nehalem-EX processors: 4096 cores (2048 cores per SSI) –8-cores per socket, 2 hardware threads per core, 4 flops/clock, 24MB L3, Turbo Boost, QPI –4 memory channels per socket strong memory bandwidth –x86 instruction set with SSE 4.2 excellent portability and ease of use SUSE Linux operating system –supports OpenMP, p-threads, MPI, PGAS models high programmer productivity –supports a huge number of ISV applications high end user productivity
5 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Programming Models & Languages UV supports an extremely broad range of programming models and languages for science, engineering, and computer science –Parallelism Coherent shared memory: OpenMP, POSIX threads (“p-threads”), OpenMPI, q-threads Distributed shared memory: UPC, Co-Array Fortran* Distributed memory: MPI, Charm++ Linux OS and standard languages enable users’ domain-specific languages, e.g. NESL –Languages C, C++, Java, UPC, Fortran, Co-Array Fortran* R, R-MPI Python, Perl, … → Rapidly express algorithms that defy distributed-memory implementation. → To existing codes, offer TB memory and high concurrency. * pending F2008-compliant compilers
6 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Cache coherency protocols ensure that all data is maintained consistently in all levels of the memory hierarchy. The unit of consistency should match the processor, i.e. one cache line. Hardware support is required to this maintain memory consistency at acceptable speeds. ccNUMA memory (a brief review; 1) ccNUMA: cache-coherent non-uniform memory access Memory is organized into a non-uniform hierarchy, where each level takes longer to access: registers1clock L1 cache, ~32 kB per core~4clocks L2 cache, ~ kB per core~11clocks L3 cache, ~1-3 MB per core, shared between cores~40clocks DRAM attached to a processor (“socket”)O(200)clocks DRAM attached to a neighboring processor on the nodeO(200)clocks DRAM attached to processors on other nodesO(1500)clocks 1 socket ~2-4 sockets many sockets
7 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Blacklight Architecture: Blade “node pair” NUMAlink-5 “node” UV Hub Intel Nehalem EX-8 QPI 64 GB RAM RAM RAM RAM UV Hub Intel Nehalem EX-8 QPI 64 GB RAM RAM RAM RAM Topology fat tree, spanning all 4096 cores Per SSI: 128 sockets 2048 cores 16 TB hardware-enabled coherent shared memory Full system: 256 sockets 4096 cores 32 TB PGAS, MPI, or hybrid parallelism NL5
8 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 I/O and Grid /bessemer –PSC’s Center-wide Lustre filesystem $SCRATCH: Zest-enabled –high efficiency scalability (designed for O(10 6 ) cores), low-cost commodity components, lightweight software layers, end-to-end parallelism, client-side caching and software parity, and a unique model of load-balancing outgoing I/O onto high- speed intermediate storage followed by asynchronous reconstruction to a 3rd-party parallel file system Gateway ready: Gram5, GridFTP, comshell, Lustre-WAN… P. Nowoczynski, N. T. B. Stone, J. Yanovich, and J. Sommerfield, Zest Checkpoint Storage System for Large Supercomputers, Petascale Data Storage Workshop ’08. papers/Nowoczynski_Zest_paper_PDSW08.pdf
9 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Memory-Intensive Analysis Use Cases Algorithm Expression –Implement algorithms and analyses, e.g. graph-theoretical, for which distributed-memory implementations have been elusive or impractical. –Enable rapid, innovative analyses of complex networks. Interactive Analysis of Large Datasets –Example: fit the whole ClueWeb09 corpus into RAM to enable development of rapid machine-learning algorithms for inferring relationships. –Foster totally new ways of exploring large datasets. Interactive queries and deeper analyses limited only by the community’s imagination.
10 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 User Productivity Use Cases Rapid Prototyping –Rapid development of algorithms for large-scale data analysis –Rapid development of “one-off” analyses –Enable creativity and exploration of ideas Familiar Programming Languages –Java, R, Octave, etc. –Leverage tools that scientists, engineers, and computer scientists already know and use. Lower the barrier to using HPC. ISV Applications –ADINA, Gaussian, VASP, … Vast memory accessible from even a modest number of cores –Leverage tools that scientists, engineers, and computer scientists already know and use. Lower the barrier to using HPC.
11 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Data crisis: genomics DNA sequencing machine throughput increasing at a rate of 5x per year Hundreds of petabytes of data will be produced in the next few years Moving and analyzing these data will be the major bottleneck in this field
12 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Genomics analysis: two basic flavors Loosely-coupled problems Sequence alignment: Read many short DNA sequences from disk and map to a reference genome –Lots of disk I/O –Fits well with MapReduce framework Tightly-coupled problems De novo assembly: Assemble a complete genome from short genome fragments generated by sequencers –Primarily a large graph problem –Works best with a lot of shared memory
13 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Sequence Assembly of Sorghum Sarah Young and Steve Rounsley (University of Arizona) Sequence assemblies of this type will be key to the iPlant Collaborative. Larger plant assemblies are planned in the future. PSC Blacklight: EARLY illumination Tested various genomes, assembly codes, and parameters to determine best options for plant genome assemblies Performed assembly of a 600+ Mbase genome of a member of the Sorghum genus on Blacklight using ABySS.
14 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 What can a machine with 16 TB shared memory do for genomics? Exploring efficient solution of both loosely and tightly-coupled problems: Sequence alignment: –Experimenting with use of ramdisk to alleviate I/O bottlenecks and increase performance –Configuring Hadoop to work on large shared memory system –Increasing productivity by allowing researchers to use simple, familiar MapReduce framework De novo assembly of huge genomes: –Human genome with 3 gigabases (Gb) of DNA typically requires GB RAM to assemble –Cancer research requires hundreds of these assemblies –Certain important species, e.g. Loblolly pine, have genomes ~10x larger than humans requiring terabytes of RAM to assemble –Metagenomics (sampling unknown microbial populations): no theoretical limit to how many base pairs one might assemble together (100x more than human assembly!) Pinus taeda (Loblolly Pine)
15 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Thermodynamic Stability of Quasicrystals Max Hutchinson and Mike Widom (Carnegie Mellon University) A leading proposal for the thermodynamic stability of quasicrystals depends on the configurational entropy associated with tile flips (“phason flips”). Exploring the entropy of symmetry-broken structures whose perimeter is an irregular octagon will allow an approximate theory of quasicrystal entropy to be developed, replacing the actual discrete tilings with a continuum system modeled as a dilute gas of interacting tiles. Quasicrystals are modeled by rhombic/octagonal tilings, for which enumeration exposes thermodynamic properties. Breadth-first search over a graph that grows super- exponentially with system size; very little locality. Nodes must carry arbitrary-precision integers. PSC Blacklight: EARLY illumination T(7) = graph for the 3,3,3,3 quasicrystal T(1)=8
16 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Performance Profiling of Million-core Runs Sameer Shende (ParaTools and University of Oregon) Metadata Information about 1million core profile datasets, TAUParaProf Manager Window. Execution Time Breakdown of LS3DF subroutines over all MPI ranks. LS3DF Routines Profiling Data on rank 1,048,575. Histogram of MPI_Barrier, distribution of the routine calls over the execution time. ~500 GB of shared memory successfully applied to the visual analysis of very large scale performance profiles, using TAU. Profile data: synthetic million-core dataset assembled from 32k-core LS3DF runs on ANL’s BG/P. PSC Blacklight: EARLY illumination
17 © 2010 Pittsburgh Supercomputing Center SG-WG Update | Sanielevici | March 18, 2011 Summary On PSC’s Blacklight resource, hardware-supported cache-coherent shared memory is enabling new data-intensive and memory- intensive analytics and simulations. In particular, Blacklight is: –enabling new kinds of analyses on large data, –bringing new communities into HPC, and –increasing the productivity of both “traditional HPC” and new users. PSC is actively working with the research community to bring this new analysis capability to diverse fields of research. This will entail development of data-intensive workflows, new algorithms, scaling and performance engineering, and software infrastructure. Interested? Contact
© 2012 Pittsburgh Supercomputing Center Big Memory = New Science Jim Kasdorf Director of Special Projects HPC User Forum Höchstleistungsrechenzentrum.
The first-generation Cell Broadband Engine (BE) processor is a multi-core chip comprised of a 64-bit Power Architecture processor core and eight synergistic.
Multiprocessing and NUMA. What we sort of assumed so far… Northbridge connects CPU and memory to rest of system – Memory controller implemented in Northbridge.
Introduction to cloud computing Jiaheng Lu Department of Computer Science Renmin University of China
Parallel I/O from a Users Perspective HPC Advisory Council Stanford University, Dec. 6, 2011 Katie Antypas Group Leader, NERSC User Services.
Page 1 LAITS Laboratory for Advanced Information Technology and Standards Duh 7/10/03 The GMU Geospatial Grid Technology Development and Application Project.
Distributed Computing Dr. Eng. Ahmed Moustafa Elmahalawy Computer Science and Engineering Department.
Next Generation Genomics: Petascale data in the life sciences Guy Coates Wellcome Trust Sanger Institute
Analysis of Computer Algorithms. What is Algorithm? Algorithm is any well-defined computational procedure that takes some value, or set of values, as.
What is an Operating System? A program that acts as an intermediary between a user of a computer and the computer hardware. Operating system goals: Execute.
Vipul Patel Ideas … Please …
Ahmad Aljebaly Department of Computer Science Western Michigan University.
2 Google GFS Bigtable Mapreduce Yahoo Hadoop.
3 Google GFS Bigtable Mapreduce Yahoo Hadoop.
GPW2005 GGF Techniques for Monitoring Large Loosely-coupled Cluster Jobs Brian L. Tierney Dan Gunter Distributed Systems Department Lawrence Berkeley National.
Why everything I learned at Leeds in 1972 is no longer true! Andrew Herbert 30 th March 2007.
2010 FutureGrid User Advisory Meeting Architecture Roadmap Long term vision 10:00-10:45, Monday, August 2, 2010 Pittsburgh, PA Gregor von Laszewski Representing.
KIT – The cooperation of Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH) The OpenCirrus TM Project: A global Testbed for Cloud.
Copyright 2004 Bernd Brügge TUM Software Engineering WS TUM System Design II Bernd Brügge Technische Universität München Applied Software Engineering.
Fusion: The App of our Eyes David Keyes, Reporter and Questioner On behalf of your CS and Applied Math colleagues ISOFS Workshop San Diego, CA 18 September.
TeraGrid Science Gateways Nancy Wilkins-Diehr TeraGrid Area Director for Science Gateways University of Michigan CI Days, November 2,
© 2012 IBM Corporation January 19, 2014 The Big Deal About Big Data Dean Compher Data Management Technical Professional for UT, NV
Multiple Processor Systems Bits of Chapters 4, 10, 16 Operating Systems: Internals and Design Principles, 6/E William Stallings.
Distributed Processing, Client/Server and Clusters Chapter 16.
Windows Azure for Research Roger Barga, Architect Cloud Computing Futures, MSR.
Cloud Computing Cloud Computing Overview of Distributed Computing.
1 Towards an Open Service Framework for Cloud-based Knowledge Discovery Domenico Talia ICAR-CNR & UNIVERSITY OF CALABRIA, Italy Cloud.
© 2016 SlidePlayer.com Inc. All rights reserved.