We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byHeidi Amis
Modified about 1 year ago
The Development of Mellanox - NVIDIA GPUDirect over InfiniBand A New Model for GPU to GPU Communications Gilad Shainer
© 2010 MELLANOX TECHNOLOGIES Overview 2 The GPUDirect project was announced Nov 2009 “NVIDIA Tesla GPUs To Communicate Faster Over Mellanox InfiniBand Networks” GPUDirect was developed by Mellanox and NVIDIA New interface (API) within the Tesla GPU driver New interface within the Mellanox InfiniBand drivers Linux kernel modification GPUDirect availability was announced May 2010 “Mellanox Scalable HPC Solutions with NVIDIA GPUDirect Technology Enhance GPU-Based HPC Performance and Efficiency” &rec_id=427http://www.mellanox.com/content/pages.php?pg=press_release_item &rec_id=427
© 2010 MELLANOX TECHNOLOGIES Why GPU Computing? 3 GPUs provide cost effective way for building supercomputers Dense packaging of compute flops with high memory bandwidth
© 2010 MELLANOX TECHNOLOGIES 4 4 The InfiniBand Architecture Industry standard defined by the InfiniBand Trade Association Defines System Area Network architecture Comprehensive specification: from physical to applications Architecture supports Host Channel Adapters (HCA) Target Channel Adapters (TCA) Switches Routers Facilitated HW design for Low latency / high bandwidth Transport offload Processor Node InfiniBand Subnet Gateway HCA Switch Processor Node HCA TCA Storage Subsystem Consoles TCA RAID Ethernet Gateway Fibre Channel HCA Subnet Manager
© 2010 MELLANOX TECHNOLOGIES Bandwidth per direction (Gb/s) 40G-IB-DDR 60G-IB-DDR 120G-IB-QDR 80G-IB-QDR 200G-IB-EDR 112G-IB-FDR 300G-IB-EDR 168G-IB-FDR 8x HDR 12x HDR # of Lanes per direction Per Lane & Rounded Per Link Bandwidth (Gb/s) 5G-IB DDR 10G-IB QDR 14G-IB-FDR (14.025) 26G-IB-EDR ( ) x12 x8 x x NDR 8x NDR 40G-IB-QDR 100G-IB-EDR 56G-IB-FDR 20G-IB-DDR 4x HDR x4 4x NDR 10G-IB-QDR 25G-IB-EDR 14G-IB-FDR 1x HDR 1x NDR Market Demand InfiniBand Link Speed Roadmap 5
© 2010 MELLANOX TECHNOLOGIES Efficient HPC 6 GPUDirect CORE-Direct Offloading Congestion Control Adaptive Routing Management Messaging Accelerations Advanced Auto-negotiation Advanced Auto-negotiation MPI
© 2010 MELLANOX TECHNOLOGIES GPU – InfiniBand Based Supercomputers GPU-InfiniBand architecture enables cost/effective supercomputers Lower system cost, less space, lower power/cooling costs 7 National Supercomputing Centre in Shenzhen (NSCS) 5K end-points (nodes) Mellanox IB – GPU PetaScale Supercomputer (#2 on the Top500)
© 2010 MELLANOX TECHNOLOGIES GPUs – InfiniBand Balanced Supercomputers GPUs introduce higher demands on the cluster communication 8 NVIDIA “Fermi” 512 cores Network Data CPU GPU Must be Balanced Must be Balanced
© 2010 MELLANOX TECHNOLOGIES System Architecture The InfiniBand Architecture (IBA) is an industry-standard fabric designed to provide high bandwidth, low-latency computing, scalability for ten-thousand nodes and multiple CPU/GPU cores per server platform and efficient utilization of compute processing resources. 40Gb/s of bandwidth node to node Up to 120Gb/s between switches Latency of 1μsec 9
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) For GPU communications “pinned” buffers are used A section in the host memory dedicated for the GPU Allows optimizations such as write-combining and overlapping GPU computation and data transfer for best performance InfiniBand uses “pinned” buffers for efficient RDMA Zero-copy data transfers, Kernel bypass 10 CPU GPU Chip set GPUMemory InfiniBand SystemMemory
© 2010 MELLANOX TECHNOLOGIES GPU-InfiniBand Bottleneck (pre-GPUDirect) The CPU has to be involved GPU to GPU data path For example: memory copies between the different “pinned buffers” Slow down the GPU communications and creates communication bottleneck 11 CPU GPU Chip set GPUMemory InfiniBand SystemMemory CPU GPU Chip set GPUMemory InfiniBand SystemMemory Transmit Receive
© 2010 MELLANOX TECHNOLOGIES 12 Mellanox – NVIDIA GPUDirect Technology Allows Mellanox InfiniBand and NVIDIA GPU to communicate faster Eliminates memory copies between InfiniBand and GPU Mellanox-NVIDIA GPUDirect Enables Fastest GPU-to-GPU Communications GPUDirect No GPUDirect
© 2010 MELLANOX TECHNOLOGIES GPUDirect Elements Linux Kernel modifications Support for sharing pinned pages between different drivers Linux Kernel Memory Manager (MM) allows NVIDIA and Mellanox drivers to share the host memory –Provides direct access for the latter to the buffers allocated by NVIDIA CUDA library, and thus, providing Zero Copy of data and better performance NVIDIA driver Allocated buffers by the CUDA library are managed by the NVIDIA Tesla driver Modifications to mark these pages to be shared so the Kernel MM will allows the Mellanox InfiniBand drivers to access them and use them for transportation without the need for copying or re-pinning them. Mellanox driver The InfiniBand application running over Mellanox InfiniBand adapters can transfer the data that resides on the buffers allocated by the NVIDIA Tesla driver Modifications to query this memory and to be able to share it with the NVIDIA Tesla drivers using the new Linux Kernel MM API Mellanox driver registers special callbacks to allow other drivers sharing the memory to notify any changes performed during run time in the shared buffers state, in order for the Mellanox driver to use the memory accordingly and to avoid invalid access to any shared pinned buffers 13
© 2010 MELLANOX TECHNOLOGIES Accelerating GPU Based Supercomputing Fast GPU to GPU communications Native RDMA for efficient data transfer Reduces latency by 30% for GPUs communication 14 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 CPU GPU Chip set GPUMemory InfiniBand System Memory 1 1 Transmit Receive
© 2010 MELLANOX TECHNOLOGIES Applications Performance - Amber Amber is a molecular dynamics software package One of the most widely used programs for bimolecular studies Extensive user base Molecular dynamics simulations calculate the motion of atoms Benchmarks FactorIX - simulates the blood coagulation factor IX, consists of 90,906 atoms Cellulose - simulates a cellulose fiber, consists of 408,609 atoms Using the PMEMD simulation program for Amber Benchmarking system – 8-node system Mellanox ConnectX-2 adapters and InfiniBand switches Each node includes one NVIDIA Fermi C2050 GPU 15 FactorIX Cellulose
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 33% performance increase with GPUDirect Performance benefit increases with cluster size 16
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect Cellulose Benchmark 29% performance increase with GPUDirect Performance benefit increases with cluster size 17
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 20% performance increase with GPUDirect Performance benefit increases with cluster size 18
© 2010 MELLANOX TECHNOLOGIES Amber Performance with GPUDirect FactorIX Benchmark 25% performance increase with GPUDirect Performance benefit increases with cluster size 19
© 2010 MELLANOX TECHNOLOGIES Summary GPUDirect enables the first phase of direct GPU-Interconnect connectivity Essential step towards efficient GPU Exa-scale computing Performance benefits range depending on application and platform From 5-10% for Linpack, to 33% for Amber Further testing will include more applications/platforms The work presented was supported by the HPC Advisory Council World-wide HPC organization (160 members) 20
21 HPC Advisory Council Members
+ William Stallings Computer Organization and Architecture 9 th Edition.
The first-generation Cell Broadband Engine (BE) processor is a multi-core chip comprised of a 64-bit Power Architecture processor core and eight synergistic.
Low Latency Networking Glenford Mapp Digital Technology Group Computer Laboratory
Network Services for Enhanced Cloud Computing T. V. Lakshman Bell Labs (Jointly with F. Hao, S. Mukherjee, H. Song)
Virtualization Technique Virtualization Technique System Virtualization I/O Virtualization.
Computer Graphics Prof. Muhammad Saeed Dept. of Computer Science & IT Federal Urdu University of Arts, Sciences and Technology.
1 Trial Lecture The Use of GPUs for High-Performance Computing 12. October 2010 Magnus Jahre.
Computer Graphics Prof. Muhammad Saeed. 2 Hardware ( Graphic Cards ) II Hardware II Computer Graphics 1 August 2012.
Silberschatz, Galvin and Gagne ©2010 Operating System Concepts Essentials – 8 th Edition Chapter 16: Windows 7.
Objectives: Chapter 7: Intranet LAN Design * Goals and considerations in LAN design * Understand the steps in systematic LAN design * Design issues associated.
What is an Operating System? A program that acts as an intermediary between a user of a computer and the computer hardware. Operating system goals: Execute.
Distributed Computing Dr. Eng. Ahmed Moustafa Elmahalawy Computer Science and Engineering Department.
MicroKernel Pattern Presented by Sahibzada Sami ud din Kashif Khurshid.
Network-I/O Convergence in Too Fast Networks: Threats and Countermeasures David R. Cheriton Stanford University.
Ahmad Aljebaly Department of Computer Science Western Michigan University.
HSA FOR THE COMMON MAN Vinod Tipparaju Heterogeneous System Software, AMD Lee Howes Heterogeneous System Software, AMD.
Data-Awareness and Low- Latency on the Enterprise Grid Getting the Most out of Your Grid with Enterprise IMDG Shay Hassidim Deputy CTO Oct 2007.
Multiprocessing and NUMA. What we sort of assumed so far… Northbridge connects CPU and memory to rest of system – Memory controller implemented in Northbridge.
1. Interconnection Networks ◦ Terminology, basics, examples Optical Interconnects ◦ Motivation ◦ Architecture examples for Data Centers and HPC ◦
The Client/Server Database Environment CS263 Lecture 12.
The Intelligent, Automated IT Infrastructure. 2 EQUALLOGIC IS CHANGING THE WAY PEOPLE EXPERIENCE STORAGE By delivering dynamic virtual storage solutions.
1 Making Parallel Processing on Clusters Efficient, Transparent and Easy for Programmers Andrzej M. Goscinski School of Computing and Mathematics Deakin.
Sreenath Chary IBM Systems Group-System z Business Unit Executive Managing TCO with SOA: Platform Makes the Difference!
Branch Repeater 5.6, 5.7 & VPX Technical Presentation.
Presented By : Vibhuti Dhiman. Outline : 1.Paper Abstract 2. Introduction 3.Design and Implementation 3.1 Event Propagation 3.2 HyperFlow Controller Application.
MapuSoft Technologies Presentation OS Abstractor, OS Changer, OS PAL and MapuSoft are registered trademarks of MapuSoft Technologies Inc. All other trademarks.
NETWORKING COMPONENTS ASSIGNMENT 3 CREATED BY JANICE THOMPSON Instructor: James West Course: 4550.
Copyright 2011 John Wiley & Sons, Inc Business Data Communications and Networking 11th Edition Jerry Fitzgerald and Alan Dennis John Wiley & Sons, Inc.
© 2016 SlidePlayer.com Inc. All rights reserved.