Download presentation
Presentation is loading. Please wait.
1
Multi-GPU System Design with Memory Networks
Gwangsun Kim, Minseok Lee, Jiyun Jeong, John Kim Department of Computer Science Korea Advanced Institute of Science and Technology
2
Single-GPU Programming Pattern
Device Memory Host Memory Data
3
Multi-GPU Programming Pattern
… Problems: 1. Programming can be challenging. 2. Inter-GPU communication cost is high. Device Memory How to place the data? A. Split B. Duplicate Host Memory Data 1 1 2 3 4 5 6 7 … 2 4 3 5 7 … 6
4
Hybrid Memory Cube (HMC)
Vault controller … Vault controller Vault DRAM layers Intra-HMC Network Logic layer I/O port … I/O port High-speed link Packet
5
Memory Network Memory network for multi-CPU [Kim et al., PACT’13] GPU
Logic layer High-speed link DRAM layers I/O port … Vault controller Intra-HMC Network GPU GPU CPU
![Memory Network Memory network for multi-CPU [Kim et al., PACT’13] GPU](http://slideplayer.com/slide/3986883/13/images/5/Memory+Network+Memory+network+for+multi-CPU+%5BKim+et+al.%2C+PACT%E2%80%9913%5D+GPU.jpg "Logic layer. High-speed link. DRAM layers. I/O port. … Vault controller. Intra-HMC Network. GPU. GPU. CPU.")
6
… Related Work NVLink for Nvidia Pascal architecture …
Drawback: some processor bandwidth dedicated to NVLink. SLI (Nvidia) and Crossfire (AMD) Graphics only. Unified virtual addressing from Nvidia Easy access to other GPU’s memory Restriction in memory allocation. MEM GPU … IO Hub CPU PCIe Switches … NVLink
7
Contents Motivation Related work Inter-GPU communication
Scalable kernel execution (SKE) GPU memory network (GMN) design CPU-GPU communication Unified memory network (UMN) Overlay network architecture Evaluation Conclusion
8
GPU Memory Network Advantage
288 GB/s 15.75 GB/s PCIe PCIe (optional) GPU Memory Network Device Memory Separate physical address spaces Unified physical address space
9
Scalable Kernel Execution (SKE)
Executes an unmodified kernel on multiple GPUs. GPUs need to support partial execution of a kernel. Kernel Partitioned Kernel Kernel Original Kernel Virtual GPU GPU GPU GPU GPU GPU Source transformation [Kim et al., PPoPP’11] [Lee et al., PACT’13] [Cabezas et al., PACT’14] Single-GPU Multi-GPU with SKE
10
Scalable Kernel Execution Implementation
Thread block 1D Kernel Thread Block range for GPU 0 Block range for GPU 1 Block range for GPU 2 ... Original kernel meta data + Block range Virtual GPU command queue Virtual GPU Application (unmodified single-GPU version) Original kernel meta data SKE Runtime GPU command queue GPU0 GPU1 …
11
Memory Address Space Organization
GPU virtual address space Fine-grained interleaving Load-balanced … Cache line 0 Cache line 1 Cache line 2 Cache line 3 Cache line 4 Cache line 5 Page A GPU X GPU Page B GPU Y Non-minimal path Page C GPU Z … … Minimal path …
12
Multi-GPU Memory Network Topology
Load-balanced GPU channels Remove path diversity among local HMCs 2D Flattened butterfly w/o concentration [ISCA’07] (FBFLY) GPU HMC 50% 43% 33% Load-balanced Distributor-based flattened butterfly [PACT’13] (dFBFLY) Sliced flattened butterfly (sFBFLY)
13
Contents Motivation Related work Inter-GPU communication
Scalable kernel execution (SKE) GPU memory network (GMN) design CPU-GPU communication Unified memory network (UMN) Overlay network architecture Evaluation Conclusion
14
Data Transfer Overhead
CPU GPU Problems: 1. CPU-GPU communication BW is low. 2. Data transfer (or memory copy) overhead. PCIe Data 1 2 … Low BW Host memory Device memory
15
Unified Memory Network
Remove PCIe bottleneck between CPU and GPUs. Eliminate memory copy between CPU and GPUs! Unified Memory Network GPU Memory Network CPU … … … … … … … … … … GPU GPU GPU … PCIe Switches IO Hub
16
Overlay Network Architecture
CPUs are latency-sensitive. GPUs are bandwidth-sensitive. Off-chip link On-chip pass-thru path [PACT’13, FB-DIMM spec.] CPU GPU
17
Methodology GPGPU-sim version 3.2 Assume SKE for evaluation
Configuration 4 HMCs per CPU/GPU 8 bidirectional channels per CPU/GPU/HMC PCIe BW: GB/s, latency: 600 ns HMC: 4 GB, 8 layers, 16 vaults, 16 banks/vault, FR-FCFS Assume 1CPU-4GPU unless otherwise mentioned. Abbreviation Configuration PCIe PCIe-based system with memcpy GMN GPU memory network-based system with memcpy UMN Unified memory network-based system (no copy)
18
SKE Performance with Different Designs
Results for selected workloads 82% reduction Compute-intensive Data-intensive *Lower is better
19
Impact of Removing Path Diversity b/w Local HMCs
14% higher 9% lower <1% diff. *Lower is better
20
Input size not large enough
Scalability 13.5x # GPUs Compute-intensive Input size not large enough *Higher is better
21
Conclusion We addressed two critical problems in multi-GPU systems with memory networks. Inter-GPU communication Improved bandwidth with GPU memory network Scalable Kernel Execution Improved Programmability CPU-GPU communication Unified memory network Eliminate data transfer Overlay network architecture Our proposed designs improve both performance and programmability of multi-GPU systems.
Similar presentations
