Low-Latency Accelerated Computing on GPUs Han Vanholder, Peter Messmer Dr. Christoph Angerer DevTech, NVIDIA

Accelerated Computing High Performance & High Energy Efficiency for Throughput Tasks CPU Serial Tasks GPU Accelerator Parallel Tasks

Accelerating Insights GOOGLE DATACENTER 1,000 CPU Servers 2,000 CPUs • 16,000 cores 600 kWatts $5,000,000 STANFORD AI LAB 3 GPU-Accelerated Servers 12 GPUs • 18,432 cores 4 kWatts $33,000 Accelerating Insights The next obvious question is whether that success could scale out In 2012 Google and Andrew Ng at Stanford published a paper showing the ability to extract high level features from a dataset of 10 million images harvested from YouTube, using a large neural network model with 1B parameters The community was impressed yet sad – only Google has the resources to build a 16 thousand core cluster for these sorts of experiments, millions of $ The following year the same group at Stanford collaborating with NVIDIA showed that they could replicate the Google results from the prior year, but using only 12 GPUs.  Total cost - $33K, smaller footprint, lower power Details below… Same time to complete, same setup of 10M 200x200 images from YouTube, 1B parameters (we actually scaled to 11 B parameters). The 25 EFLOP model recognizes 1000 classes. Humans recognize somewhere around 10,000-100,000 classes. Average adult vocabulary is 20-35,000 words (kids are much smaller), and we each can recognize about 10,000 faces. We can recognize places as well. Add them all up, I'd guess most people are somewhere between 10K and 100K classes. These servers for Stanford & Google were for *training* the model.  In addition, there’s also the computational requirements for doing the actual *classification* for every new image Training one model today is 10s of EFLOPs (note the NYU model took ~25 EXAFLOPs to train) Processing an image costs only ~TFLOPs per image – but since over 1 billion photos are uploaded to the internet every day, the collective cost is huge = 46 PFLOP/s (ORNL Titan is only 27 PFLOP/s) 47 PFLOP/s = ~23,000 GPUs are required to keep up in real-time with images uploaded to the internet That’s just with today’s models For a larger “vocabulary” (more classes), the training set grows, and the computational requirements increase for training Also, classification isn’t the only think you want to know – you also want to know things like “where is the object in the image” (localization), “what are all of the objects in the image” (detection), or “who is that” (recognition).  All this drives up the cost of the real-time image processing above 46 PFLOP/s All of these emerging requirements will continue to drive up the demand for computational resources to keep up with the data deluge A few caveats: The 46 PFLOP/s (SP) number that I calculated for these 23,148 GPUs depends on how many FLOPs/s you assume are achieved on the GPU for this operation, so it might be off by ~2x.  CEO math is reasonable for 46 PFLOP/s if you assume 2 TFLOP/s sustained single precision Most people will think 17.6 PFLOP/s double precision for LINPACK for ORNL Titan.  The 27 PFLOP/s I quoted below as for peak (not LINPACK).  In single precision Titan might be ~2x that number – so ~35 PFLOP/s single precision Now You Can Build Google’s $1M Artificial Brain on the Cheap “ Deep learning with COTS HPC systems, A. Coates, B. Huval, T. Wang, D. Wu, A. Ng, B. Catanzaro  ICML 2013

From HPC to Enterprise Data Center Oil & Gas Higher Ed Government Supercomputing Finance Consumer Web Air Force Research Laboratory Tokyo Institute of Technology Naval Research Laboratory

Tesla: Platform for Accelerated Datacenters Data Center Infrastructure Development Optimized Systems Communication Solutions Infrastructure Management Programming Languages Development Tools Software Applications Partner Ecosystem Tesla is more than just great GPU Accelerator Boards. It combines the world’s fastest GPU accelerators, the widely used CUDA parallel computing model, enterprise-class support and maintenance services, and a comprehensive ecosystem of software developers, software vendors, and datacenter system OEMs to deliver unrivalled performance. It combines Fast GPU Accelerators, advanced system management features, accelerated communication technology, and is supported by popular infrastructure management software. The Tesla platform provides computing professionals the tools to easily build, test and deploy accelerated applications in the datacenter. GPU Accelerators Interconnect System Management Compiler Solutions Profile and Debug Libraries Enterprise Services Tesla Accelerated Computing Platform

Common Programming Models Across Multiple CPUs Libraries Programming Languages Compiler Directives AmgX cuBLAS / The other primary goal for us is to provide an open platform that supports all CPU archtectures, industry standards. NVIDIA is the only processor company in the world with the open platform to work with any CPU architecture, OpenPOWER, ARM, or x86. Intel can’t do this. They are tied to x86. In fact, Intel & x86 is becoming more “closed”, own network stack. Our vision is OEMs will build what is right for their customers, the right CPU, etc, and GPUs will be there to accelerate their work. We want to make sure when our customers write GPU code, …, it will run without change on all three platforms. X86

Normalized Performance GPU Roadmap 20 Pascal Unified Memory 3D Memory NVLink 16 12 Maxwell DX12 Normalized Performance 8 Kepler Dynamic Parallelism 4 Fermi FP64 Tesla CUDA 2008 2010 2012 2014 2016

GPUDirect

Multi-GPU: Unified Virtual Addressing Single Partitioned Address Space System Memory GPU0 Memory GPU1 Memory 0x0000 0xFFFF CPU GPU0 GPU1 PCI-e

GPUDirect Technologies GPUDirect Shared GPU-Sysmem for Inter-node copy optimization How: Use GPUDirect-aware 3rd party network drivers GPUDirect P2P Transfers for on-node GPU-GPU memcpy How: Use CUDA APIs directly in application How: use P2P-aware MPI implementation GPUDirect P2P Access for on-node inter-GPU LD/ST access How: Access remote data by address directly in GPU device code GPUDirect RDMA for Inter-node copy optimization What: 3rd party PCIe devices can read and write GPU memory How: Use GPUDirect RDMA-aware 3rd party network drivers and MPI implementations or custom device drivers for other hardware

GPUDirect P2P: GPU-GPU Direct Access PCIe PCIe GPU Memory GPU Memory CPU/IOH GPUDirect v2.0 enables GPUs to communicate directly, both during cudaMemcpyD2D and from within CUDA kernels CUDA 4.0 feature CPU Memory

P2P Goal: Improve intra-node programming model Improve CUDA programming model How? Transfer data between two GPUs quickly/easily int main() { double *cpuTmp, *gpu0Data, gpu1Data; setup (gpu0Data, gpu1Data); cudaSetDevice (0); kernel <<< … >>> (gpu0Data); cudaMemcpy (cpuTmp, gpu0Data); cudaMemcpy (gpu1Data, cpuTmp); cudaSetDevice (1); }

GPUDirect P2P: Common use cases MPI implementation optimization for Intra-Node communication HPC applications that fit on a single node You get much better efficiency with GPUDirect P2P (compared to MPI) Can use between ranks with cudaIpc() APIs

GPUDirect RDMA GPU CPU/IOH Third Party Hardware PCIe PCIe GPU CPU Memory GPU GPUDirect v3.0 enables third party devices to directly access GPU memory without CPU or chipset involvement CUDA 5.0 feature Any vendor can implement with no hardware modification (only software device driver changes)! Video capture devices NICs (Infiniband, Ethernet and others) Storage devices (Future SSD + Database applications) Third Party Hardware GPU Memory

GPUDirect RDMA Goal Inter-node Latency and Bandwidth How? Transfer data between GPU and third party device (e.g. NIC) with possibly zero host-side copies int main() { double *cpuData, *cpuTmp, *gpuData; setup (gpuData); kernel <<< … >>> (gpuData); cudaDeviceSynchronize (); cudaMemcpy (cpuTmp, gpuData); memcpy (cpuData, cpuTmp); MPI_Send (gpuData) }

GPUDirect RDMA: What does it get you? Latency Reduction MPI_Send latency of 25µs with Shared GPU-Sysmem* No overlap possible Bidirectional transfer is difficult MPI_Send latency of 5µs with RDMA Does not affect running kernels Unlimited concurrency RDMA possible! MPI-3 One sided of 3µs

GPUDirect RDMA: Common use cases Inter-Node MPI communication Transfer data between GPU and a remote Node Use CUDA-aware MPI Interface with third party hardware Requires adopting GPUDirect-Interop API in vendor driver stack Limitation GPUDirect RDMA does not work with CUDA Unified Memory today

NVLINK

Pascal GPU Features NVLINK and Stacked Memory GPU high speed interconnect 80-200 GB/s 3D Stacked Memory 4x Higher Bandwidth (~1 TB/s) 3x Larger Capacity 4x More Energy Efficient per bit

NVLink Unleashes Multi-GPU Performance Over 2x Application Performance Speedup When Next-Gen GPUs Connect via NVLink Versus PCIe GPUs Interconnected with NVLink CPU PCIe Switch Speedup vs PCIe based Server Want to show that NVLink provides immediate application speed-up, simply by allowing GPUs-to-GPU to communicate faster Start at 0x0 Why faster? Most HPC applications are multi-GPU enabled For multi-GPU applications, GPU-to-GPU app TESLA GPU TESLA GPU 5x Faster than PCIe Gen3 x16 3D FFT, ANSYS: 2 GPU configuration, AMBER Cellulose (256x128x128), FFT problem size (256^3), 4 GPU configuration

NVLink High-Speed GPU Interconnect Example: 8-GPU Server with NVLink Important point is the NVLink connection between the GPUs – so GPUDirect fuctions at 10x PCIe speed

Machine Learning

Machine Learning using Deep Neural Networks Input Result Hinton et al., 2006; Bengio et al., 2007; Bengio & LeCun, 2007; Lee et al., 2008; 2009 Visual Object Recognition Using Deep Convolutional Neural Networks Rob Fergus (New York University / Facebook) http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php#2985

3 Drivers for Deep Learning Powerful GPU Accelerators More Data Better Models

Broad use of GPUs in Deep learning Early Adopters Use Cases Talks @ GTC Image Detection Face Recognition Gesture Recognition Video Search & Analytics Speech Recognition & Translation Recommendation Engines Indexing & Search There’s already quite a body of research in the public domain describing the use of GPUs for machine learning tasks NVIDIA hosts our annual GPU Technology Conference in San Jose every spring This year we had a fantastic number of talks about using CUDA for machine learning and data sciences Encourage you to attend – lots of opportunities for cross-pollination across domains, especially with machine learning   Image Analytics for Creative Cloud Speech/Image Recognition Image Classification Hadoop Recommendation Search Rankings

What is Next? Analyzing Unstructured Data Anomaly Detection Behavior Prediction Diagnostic Support “Any product that excites you over the next five years and makes you think: ‘That is magical, how did they do that?’, is probably based on this [deep learning].” Steve Jurvetson, Partner DFJ Venture Language Analysis ….

Database Acceleration Beyond Deep Learning Graph Analytics Database Acceleration Real Time Analytics Visualization of social network analysis by Martin Grandjean is licensed under CC BY-SA 3.0

Deep Learning Sessions Adobe Google Alibaba iFlytek, Ltd Baidu NUANCE Carnegie Mellon Stanford Univ Facebook UC Berkeley Flickr / Yahoo Univ of Toronto GTC 2015 had many Deep Learning Sessions Check GTC on-demand http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php

NVIDIA in OCP

Unlocking Access to the Tesla Platform Engaging with OEMs, end customers and technology partners to include NVIDIA Accelerators in the OCP Platform NVLINK based designs for maximum performance Standard PCIe designs for scale out Open Compute Project

Enabling NVLink GPU-CPU connections KEPLER GPU PASCAL GPU NVLink High-Speed GPU Interconnect NVLink POWER CPU NVLink PCIe PCIe X86, ARM64, POWER CPU X86, ARM64, POWER CPU 2014 2016

Thanks cangerer@nvidia.com