We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byHeidi Amis
Modified about 1 year ago
The Development of Mellanox - NVIDIA GPUDirect over InfiniBand A New Model for GPU to GPU Communications Gilad Shainer
© 2010 MELLANOX TECHNOLOGIES Overview 2 The GPUDirect project was announced Nov 2009 “NVIDIA Tesla GPUs To Communicate Faster Over Mellanox InfiniBand Networks” http://www.nvidia.com/object/io_1258539409179.html GPUDirect was developed by Mellanox and NVIDIA New interface (API) within the Tesla GPU driver New interface within the Mellanox InfiniBand drivers Linux kernel modification GPUDirect availability was announced May 2010 “Mellanox Scalable HPC Solutions with NVIDIA GPUDirect Technology Enhance GPU-Based HPC Performance and Efficiency” http://www.mellanox.com/content/pages.php?pg=press_release_item &rec_id=427http://www.mellanox.com/content/pages.php?pg=press_release_item &rec_id=427
© 2010 MELLANOX TECHNOLOGIES Why GPU Computing? 3 GPUs provide cost effective way for building supercomputers Dense packaging of compute flops with high memory bandwidth
© 2010 MELLANOX TECHNOLOGIES 4 4 The InfiniBand Architecture Industry standard defined by the InfiniBand Trade Association Defines System Area Network architecture Comprehensive specification: from physical to applications Architecture supports Host Channel Adapters (HCA) Target Channel Adapters (TCA) Switches Routers Facilitated HW design for Low latency / high bandwidth Transport offload Processor Node InfiniBand Subnet Gateway HCA Switch Processor Node HCA TCA Storage Subsystem Consoles TCA RAID Ethernet Gateway Fibre Channel HCA Subnet Manager
© 2010 MELLANOX TECHNOLOGIES 2005-2006-2007-2008-2009-2010-2011 Bandwidth per direction (Gb/s) 40G-IB-DDR 60G-IB-DDR 120G-IB-QDR 80G-IB-QDR 200G-IB-EDR 112G-IB-FDR 300G-IB-EDR 168G-IB-FDR 8x HDR 12x HDR # of Lanes per direction Per Lane & Rounded Per Link Bandwidth (Gb/s) 5G-IB DDR 10G-IB QDR 14G-IB-FDR (14.025) 26G-IB-EDR (25.78125) 12 60+60 120+120168+168300+300 8 40+4080+80112+112200+200 4 20+2040+4056+56100+100 1 5+510+1014+1425+25 x12 x8 x1 2014 12x NDR 8x NDR 40G-IB-QDR 100G-IB-EDR 56G-IB-FDR 20G-IB-DDR 4x HDR x4 4x NDR 10G-IB-QDR 25G-IB-EDR 14G-IB-FDR 1x HDR 1x NDR Market Demand InfiniBand Link Speed Roadmap 5
© 2010 MELLANOX TECHNOLOGIES Efficient HPC 6 GPUDirect CORE-Direct Offloading Congestion Control Adaptive Routing Management Messaging Accelerations Advanced Auto-negotiation Advanced Auto-negotiation MPI
© 2010 MELLANOX TECHNOLOGIES GPU – InfiniBand Based Supercomputers GPU-InfiniBand architecture enables cost/effective supercomputers Lower system cost, less space, lower power/cooling costs 7 National Supercomputing Centre in Shenzhen (NSCS) 5K end-points (nodes) Mellanox IB – GPU PetaScale Supercomputer (#2 on the Top500)
© 2010 MELLANOX TECHNOLOGIES GPUs – InfiniBand Balanced Supercomputers GPUs introduce higher demands on the cluster communication 8 NVIDIA “Fermi” 512 cores Network Data CPU GPU Must be Balanced Must be Balanced
© 2010 MELLANOX TECHNOLOGIES System Architecture The InfiniBand Architecture (IBA) is an industry-standard fabric designed to provide high bandwidth, low-latency computing, scalability for ten-thousand nodes and multiple CPU/GPU cores per server platform and efficient utilization of compute processing resources. 40Gb/s of bandwidth node to node Up to 120Gb/s between switches Latency of 1μsec 9
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) For GPU communications “pinned” buffers are used A section in the host memory dedicated for the GPU Allows optimizations such as write-combining and overlapping GPU computation and data transfer for best performance InfiniBand uses “pinned” buffers for efficient RDMA Zero-copy data transfers, Kernel bypass 10 CPU GPU Chip set GPUMemory InfiniBand SystemMemory
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) The CPU has to be involved GPU to GPU data path For example: memory copies between the different “pinned buffers” Slow down the GPU communications and creates communication bottleneck 11 CPU GPU Chip set GPUMemory InfiniBand SystemMemory 1 1 2 2 CPU GPU Chip set GPUMemory InfiniBand SystemMemory 1 1 2 2 Transmit Receive
© 2010 MELLANOX TECHNOLOGIES 12 Mellanox – NVIDIA GPUDirect Technology Allows Mellanox InfiniBand and NVIDIA GPU to communicate faster Eliminates memory copies between InfiniBand and GPU Mellanox-NVIDIA GPUDirect Enables Fastest GPU-to-GPU Communications GPUDirect No GPUDirect
© 2010 MELLANOX TECHNOLOGIES GPUDirect Elements Linux Kernel modifications Support for sharing pinned pages between different drivers Linux Kernel Memory Manager (MM) allows NVIDIA and Mellanox drivers to share the host memory –Provides direct access for the latter to the buffers allocated by NVIDIA CUDA library, and thus, providing Zero Copy of data and better performance NVIDIA driver Allocated buffers by the CUDA library are managed by the NVIDIA Tesla driver Modifications to mark these pages to be shared so the Kernel MM will allows the Mellanox InfiniBand drivers to access them and use them for transportation without the need for copying or re-pinning them. Mellanox driver The InfiniBand application running over Mellanox InfiniBand adapters can transfer the data that resides on the buffers allocated by the NVIDIA Tesla driver Modifications to query this memory and to be able to share it with the NVIDIA Tesla drivers using the new Linux Kernel MM API Mellanox driver registers special callbacks to allow other drivers sharing the memory to notify any changes performed during run time in the shared buffers state, in order for the Mellanox driver to use the memory accordingly and to avoid invalid access to any shared pinned buffers 13
© 2010 MELLANOX TECHNOLOGIES Accelerating GPU Based Supercomputing Fast GPU to GPU communications Native RDMA for efficient data transfer Reduces latency by 30% for GPUs communication 14 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 Transmit Receive
© 2010 MELLANOX TECHNOLOGIES Applications Performance - Amber Amber is a molecular dynamics software package One of the most widely used programs for bimolecular studies Extensive user base Molecular dynamics simulations calculate the motion of atoms Benchmarks FactorIX - simulates the blood coagulation factor IX, consists of 90,906 atoms Cellulose - simulates a cellulose fiber, consists of 408,609 atoms Using the PMEMD simulation program for Amber Benchmarking system – 8-node system Mellanox ConnectX-2 adapters and InfiniBand switches Each node includes one NVIDIA Fermi C2050 GPU 15 FactorIX Cellulose
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 33% performance increase with GPUDirect Performance benefit increases with cluster size 16
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 29% performance increase with GPUDirect Performance benefit increases with cluster size 17
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 20% performance increase with GPUDirect Performance benefit increases with cluster size 18
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 25% performance increase with GPUDirect Performance benefit increases with cluster size 19
© 2010 MELLANOX TECHNOLOGIES Summary GPUDirect enables the first phase of direct GPU-Interconnect connectivity Essential step towards efficient GPU Exa-scale computing Performance benefits range depending on application and platform From 5-10% for Linpack, to 33% for Amber Further testing will include more applications/platforms The work presented was supported by the HPC Advisory Council http://www.hpcadvisorycouncil.com/ World-wide HPC organization (160 members) 20
21 HPC Advisory Council Members
Mellanox Connectivity Solutions for Scalable HPC Highest Performing, Most Efficient End-to-End Connectivity for Servers and Storage September 2010 Brandon.
Mellanox Connectivity Solutions for Scalable HPC Highest Performing, Most Efficient End-to-End Connectivity for Servers and Storage April 2010.
Gilad Shainer, VP of Marketing Dec 2013 Interconnect Your Future.
©2009 HP Confidential template rev Ed Turkel Manager, WorldWide HPC Marketing 4/7/2011 BUILDING THE GREENEST PRODUCTION SUPERCOMPUTER IN THE.
Maximizing The Compute Power With Mellanox InfiniBand Connectivity Gilad Shainer Wolfram Technology Conference 2006.
Remote Direct Memory Access (RDMA) over IP PFLDNet 2003, Geneva Stephen Bailey, Sandburst Corp., Allyn Romanow, Cisco Systems,
The NE010 iWARP Adapter Gary Montry Senior Scientist
Scheduling Strategies for HPC as a Service (HPCaaS) for Bio-Science Applications Sep 2009 High Performance Interconnects for Distributed Computing (HPI-DC)
© 2012 MELLANOX TECHNOLOGIES 1 The Exascale Interconnect Technology Rich Graham – Sr. Solutions Architect.
NVIDIA Kepler GK110 Architecture (chip) (As used in coit-grid08.uncc.edu K20 GPU server) Highlights – To discuss in class Extracted directly from: “Whitepaper.
Introduction to Data Analysis with R on HPC Texas Advanced Computing Center Feb
Current major high performance networking technologies InfiniBand 10G-Ethernet.
InfiniSwitch Company Confidential. 2 InfiniSwitch Agenda InfiniBand Overview Company Overview Product Strategy Q&A.
Performance Analysis of Virtualization for High Performance Computing A Practical Evaluation of Hypervisor Overheads Matthew Cawood University of Cape.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Update IDC HPC Forum.
IWARP Ethernet Key to Driving Ethernet into the Future Brian Hausauer Chief Architect NetEffect, Inc.
Taking the Complexity out of Cluster Computing Vendor Update HPC User Forum Arend Dittmer Director Product Management HPC April,
1 Public DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience Arkady Kanevsky & Peter Corbett Network Appliance Vijay Velusamy.
Early Linpack Performance Benchmarking on IPE Mole-8.5 Fermi GPU Cluster Xianyi Zhang 1),2) and Yunquan Zhang 1),3) 1) Laboratory of Parallel Software.
A Practical Evaluation of Hypervisor Overheads Matthew Cawood Supervised by: Dr. Simon Winberg University of Cape Town Performance Analysis of Virtualization.
CMS week, June 2002, CERN 1 First P2P Measurements on Infiniband Luciano Berti INFN Laboratori Nazionali di Legnaro.
GPU System Architecture Alan Gray EPCC The University of Edinburgh.
An overview of Infiniband Reykjavik, June 24th 2008 R E Y K J A V I K U N I V E R S I T Y Dept. Computer Science Center for Analysis and Design of Intelligent.
Infiniband Bart Taylor. What it is InfiniBand™ Architecture defines a new interconnect technology for servers that changes the way data centers will be.
© 2012 MELLANOX TECHNOLOGIES 1 Disruptive Technologies in HPC Interconnect HPC User Forum April 16, 2012.
Swapping to Remote Memory over InfiniBand: An Approach using a High Performance Network Block Device Shuang LiangRanjit NoronhaDhabaleswar K. Panda IEEE.
Scalable Multi-Cache Simulation Using GPUs Michael Moeng Sangyeun Cho Rami Melhem University of Pittsburgh.
2006 Sonoma Workshop February 2006Page 1 Sockets Direct Protocol (SDP) for Windows - Motivation and Plans Gilad Shainer Mellanox Technologies Inc.
HPCC Mid-Morning Break High Performance Computing on a GPU cluster Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery.
Issues of HPC software From the experience of TH-1A Lu Yutong NUDT.
1 InfiniBand HW Architecture InfiniBand Unified Fabric InfiniBand Architecture Router xCA Link Topology Switched Fabric (vs shared bus) 64K nodes per sub-net.
March 9, 2015 San Jose Compute Engineering Workshop.
Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer – Intel Corporation OpenIB Maintainer Yaron Haviv CTO.
1)Leverage raw computational power of GPU Magnitude performance gains possible.
© 2013 Mellanox Technologies 1 NoSQL DB Benchmarking with high performance Networking solutions WBDB, Xian, July 2013.
GPU Programming with CUDA – Accelerated Architectures Mike Griffiths
© 2006 EMC Corporation. All rights reserved. Internet Protocol Storage Area Networks (IP SAN) Module 3.4.
IBM Power system – HPC solution Real-time control for adaptive optics workshop January 2016 Guillaume Villette
1 petaFLOPS+ in 10 racks TB2–TL system announcement Rev 1A.
SRP Update Bart Van Assche,. Overview Involvement With SRP SRP Protocol Overview Recent SRP Driver Changes Possible Future Directions March 30 – April.
2006 Sonoma Workshop February 2006Page 1 MemFree Technology Gilad Shainer Mellanox Technologies Inc.
VIA and Its Extension To TCP/IP Network Yingping Lu Based on Paper “Queue Pair IP, …” by Philip Buonadonna.
HPCC Mid-Morning Break Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery Introduction to the new GPU (GFX) cluster.
Experiences with VI Communication for Database Storage Yuanyuan Zhou, Angelos Bilas, Suresh Jagannathan, Cezary Dubnicki, Jammes F. Philbin, Kai Li.
ISER update 2014 OFA Developer Workshop Sunday, March 30 - Wednesday, April 2, 2014 Monterey CA Eyal Salomon Mellanox Technologies.
Monte-Carlo method and Parallel computing An introduction to GPU programming Mr. Fang-An Kuo, Dr. Matthew R. Smith NCHC Applied Scientific Computing.
CONFIDENTIAL Server and Storage Connectivity Solutions April 2009.
© 2017 SlidePlayer.com Inc. All rights reserved.