We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byHeidi Amis
Modified over 4 years ago
The Development of Mellanox - NVIDIA GPUDirect over InfiniBand A New Model for GPU to GPU Communications Gilad Shainer
© 2010 MELLANOX TECHNOLOGIES Overview 2 The GPUDirect project was announced Nov 2009 “NVIDIA Tesla GPUs To Communicate Faster Over Mellanox InfiniBand Networks” http://www.nvidia.com/object/io_1258539409179.html GPUDirect was developed by Mellanox and NVIDIA New interface (API) within the Tesla GPU driver New interface within the Mellanox InfiniBand drivers Linux kernel modification GPUDirect availability was announced May 2010 “Mellanox Scalable HPC Solutions with NVIDIA GPUDirect Technology Enhance GPU-Based HPC Performance and Efficiency” http://www.mellanox.com/content/pages.php?pg=press_release_item &rec_id=427http://www.mellanox.com/content/pages.php?pg=press_release_item &rec_id=427
© 2010 MELLANOX TECHNOLOGIES Why GPU Computing? 3 GPUs provide cost effective way for building supercomputers Dense packaging of compute flops with high memory bandwidth
© 2010 MELLANOX TECHNOLOGIES 4 4 The InfiniBand Architecture Industry standard defined by the InfiniBand Trade Association Defines System Area Network architecture Comprehensive specification: from physical to applications Architecture supports Host Channel Adapters (HCA) Target Channel Adapters (TCA) Switches Routers Facilitated HW design for Low latency / high bandwidth Transport offload Processor Node InfiniBand Subnet Gateway HCA Switch Processor Node HCA TCA Storage Subsystem Consoles TCA RAID Ethernet Gateway Fibre Channel HCA Subnet Manager
© 2010 MELLANOX TECHNOLOGIES 2005-2006-2007-2008-2009-2010-2011 Bandwidth per direction (Gb/s) 40G-IB-DDR 60G-IB-DDR 120G-IB-QDR 80G-IB-QDR 200G-IB-EDR 112G-IB-FDR 300G-IB-EDR 168G-IB-FDR 8x HDR 12x HDR # of Lanes per direction Per Lane & Rounded Per Link Bandwidth (Gb/s) 5G-IB DDR 10G-IB QDR 14G-IB-FDR (14.025) 26G-IB-EDR (25.78125) 12 60+60 120+120168+168300+300 8 40+4080+80112+112200+200 4 20+2040+4056+56100+100 1 5+510+1014+1425+25 x12 x8 x1 2014 12x NDR 8x NDR 40G-IB-QDR 100G-IB-EDR 56G-IB-FDR 20G-IB-DDR 4x HDR x4 4x NDR 10G-IB-QDR 25G-IB-EDR 14G-IB-FDR 1x HDR 1x NDR Market Demand InfiniBand Link Speed Roadmap 5
© 2010 MELLANOX TECHNOLOGIES Efficient HPC 6 GPUDirect CORE-Direct Offloading Congestion Control Adaptive Routing Management Messaging Accelerations Advanced Auto-negotiation Advanced Auto-negotiation MPI
© 2010 MELLANOX TECHNOLOGIES GPU – InfiniBand Based Supercomputers GPU-InfiniBand architecture enables cost/effective supercomputers Lower system cost, less space, lower power/cooling costs 7 National Supercomputing Centre in Shenzhen (NSCS) 5K end-points (nodes) Mellanox IB – GPU PetaScale Supercomputer (#2 on the Top500)
© 2010 MELLANOX TECHNOLOGIES GPUs – InfiniBand Balanced Supercomputers GPUs introduce higher demands on the cluster communication 8 NVIDIA “Fermi” 512 cores Network Data CPU GPU Must be Balanced Must be Balanced
© 2010 MELLANOX TECHNOLOGIES System Architecture The InfiniBand Architecture (IBA) is an industry-standard fabric designed to provide high bandwidth, low-latency computing, scalability for ten-thousand nodes and multiple CPU/GPU cores per server platform and efficient utilization of compute processing resources. 40Gb/s of bandwidth node to node Up to 120Gb/s between switches Latency of 1μsec 9
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) For GPU communications “pinned” buffers are used A section in the host memory dedicated for the GPU Allows optimizations such as write-combining and overlapping GPU computation and data transfer for best performance InfiniBand uses “pinned” buffers for efficient RDMA Zero-copy data transfers, Kernel bypass 10 CPU GPU Chip set GPUMemory InfiniBand SystemMemory
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) The CPU has to be involved GPU to GPU data path For example: memory copies between the different “pinned buffers” Slow down the GPU communications and creates communication bottleneck 11 CPU GPU Chip set GPUMemory InfiniBand SystemMemory 1 1 2 2 CPU GPU Chip set GPUMemory InfiniBand SystemMemory 1 1 2 2 Transmit Receive
© 2010 MELLANOX TECHNOLOGIES 12 Mellanox – NVIDIA GPUDirect Technology Allows Mellanox InfiniBand and NVIDIA GPU to communicate faster Eliminates memory copies between InfiniBand and GPU Mellanox-NVIDIA GPUDirect Enables Fastest GPU-to-GPU Communications GPUDirect No GPUDirect
© 2010 MELLANOX TECHNOLOGIES GPUDirect Elements Linux Kernel modifications Support for sharing pinned pages between different drivers Linux Kernel Memory Manager (MM) allows NVIDIA and Mellanox drivers to share the host memory –Provides direct access for the latter to the buffers allocated by NVIDIA CUDA library, and thus, providing Zero Copy of data and better performance NVIDIA driver Allocated buffers by the CUDA library are managed by the NVIDIA Tesla driver Modifications to mark these pages to be shared so the Kernel MM will allows the Mellanox InfiniBand drivers to access them and use them for transportation without the need for copying or re-pinning them. Mellanox driver The InfiniBand application running over Mellanox InfiniBand adapters can transfer the data that resides on the buffers allocated by the NVIDIA Tesla driver Modifications to query this memory and to be able to share it with the NVIDIA Tesla drivers using the new Linux Kernel MM API Mellanox driver registers special callbacks to allow other drivers sharing the memory to notify any changes performed during run time in the shared buffers state, in order for the Mellanox driver to use the memory accordingly and to avoid invalid access to any shared pinned buffers 13
© 2010 MELLANOX TECHNOLOGIES Accelerating GPU Based Supercomputing Fast GPU to GPU communications Native RDMA for efficient data transfer Reduces latency by 30% for GPUs communication 14 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 Transmit Receive
© 2010 MELLANOX TECHNOLOGIES Applications Performance - Amber Amber is a molecular dynamics software package One of the most widely used programs for bimolecular studies Extensive user base Molecular dynamics simulations calculate the motion of atoms Benchmarks FactorIX - simulates the blood coagulation factor IX, consists of 90,906 atoms Cellulose - simulates a cellulose fiber, consists of 408,609 atoms Using the PMEMD simulation program for Amber Benchmarking system – 8-node system Mellanox ConnectX-2 adapters and InfiniBand switches Each node includes one NVIDIA Fermi C2050 GPU 15 FactorIX Cellulose
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 33% performance increase with GPUDirect Performance benefit increases with cluster size 16
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 29% performance increase with GPUDirect Performance benefit increases with cluster size 17
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 20% performance increase with GPUDirect Performance benefit increases with cluster size 18
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 25% performance increase with GPUDirect Performance benefit increases with cluster size 19
© 2010 MELLANOX TECHNOLOGIES Summary GPUDirect enables the first phase of direct GPU-Interconnect connectivity Essential step towards efficient GPU Exa-scale computing Performance benefits range depending on application and platform From 5-10% for Linpack, to 33% for Amber Further testing will include more applications/platforms The work presented was supported by the HPC Advisory Council http://www.hpcadvisorycouncil.com/ World-wide HPC organization (160 members) 20
21 HPC Advisory Council Members
Issues of HPC software From the experience of TH-1A Lu Yutong NUDT.
Scalable Multi-Cache Simulation Using GPUs Michael Moeng Sangyeun Cho Rami Melhem University of Pittsburgh.
©2009 HP Confidential template rev Ed Turkel Manager, WorldWide HPC Marketing 4/7/2011 BUILDING THE GREENEST PRODUCTION SUPERCOMPUTER IN THE.
Monte-Carlo method and Parallel computing An introduction to GPU programming Mr. Fang-An Kuo, Dr. Matthew R. Smith NCHC Applied Scientific Computing.
GPU System Architecture Alan Gray EPCC The University of Edinburgh.
Early Linpack Performance Benchmarking on IPE Mole-8.5 Fermi GPU Cluster Xianyi Zhang 1),2) and Yunquan Zhang 1),3) 1) Laboratory of Parallel Software.
Performance Analysis of Virtualization for High Performance Computing A Practical Evaluation of Hypervisor Overheads Matthew Cawood University of Cape.
HPCC Mid-Morning Break High Performance Computing on a GPU cluster Dirk Colbry, Ph.D. Research Specialist Institute for Cyber Enabled Discovery.
OpenFOAM on a GPU-based Heterogeneous Cluster
1 InfiniBand HW Architecture InfiniBand Unified Fabric InfiniBand Architecture Router xCA Link Topology Switched Fabric (vs shared bus) 64K nodes per sub-net.
VIA and Its Extension To TCP/IP Network Yingping Lu Based on Paper “Queue Pair IP, …” by Philip Buonadonna.
RDMA ENABLED WEB SERVER Rajat Sharma. Objective To implement a Web Server serving HTTP client requests through RDMA replacing the traditional TCP/IP.
An overview of Infiniband Reykjavik, June 24th 2008 R E Y K J A V I K U N I V E R S I T Y Dept. Computer Science Center for Analysis and Design of Intelligent.
1 petaFLOPS+ in 10 racks TB2–TL system announcement Rev 1A.
IWARP Ethernet Key to Driving Ethernet into the Future Brian Hausauer Chief Architect NetEffect, Inc.
Scheduling Strategies for HPC as a Service (HPCaaS) for Bio-Science Applications Sep 2009 High Performance Interconnects for Distributed Computing (HPI-DC)
© 2013 Mellanox Technologies 1 NoSQL DB Benchmarking with high performance Networking solutions WBDB, Xian, July 2013.
SRP Update Bart Van Assche,.
GPU Programming with CUDA – Accelerated Architectures Mike Griffiths
Roland Dreier Technical Lead – Cisco Systems, Inc. OpenIB Maintainer Sean Hefty Software Engineer – Intel Corporation OpenIB Maintainer Yaron Haviv CTO.
© 2020 SlidePlayer.com Inc. All rights reserved.