Presentation is loading. Please wait.

Presentation is loading. Please wait.

GPGPU Architectures Martin Kruliš Martin Kruliš 11. 4. 2017.

Similar presentations


Presentation on theme: "GPGPU Architectures Martin Kruliš Martin Kruliš 11. 4. 2017."— Presentation transcript:

1 GPGPU Architectures Martin Kruliš Martin Kruliš

2 History 1996 - 3Dfx Voodoo 1 1999 - NVIDIA GeForce 256
First graphical (3D) accelerator for desktop PCs NVIDIA GeForce 256 First Transform&Lightning unit NVIDIA GeForce2, ATI Radeon GPU has programmable parts DirectX – vertex and fragment shaders (v1.0) OpenGL, DirectX 10, Windows Vista Unified shader architecture in HW Geometry shader added Martin Kruliš

3 History 2007 - NVIDIA CUDA 2007 - AMD Stream SDK (previously CTM)
First GPGPU solution, restricted to NVIDIA GPUs AMD Stream SDK (previously CTM) OpenCL, Direct Compute OpenCL revision 1.2 NVIDIA Kepler Architecture 2013 – OpenCL revision 2.0 2014 – NVIDIA Maxwell Architecture 2016 – OpenCL revision 2.2, NVIDIA Pascal 2017 – NVIDIA OpenCL 2.0 beta Martin Kruliš

4 GPU in comparison with CPU
Few cores per chip General purpose cores Processing different threads Huge caches to reduce memory latency Locality of reference problem GPU Many cores per chip Cores specialized for numeric computations SIMT thread processing Huge amount of threads and fast context switch Results in more complex memory transfers Architecture Convergence Martin Kruliš

5 NVIDIA Fermi GPU Architecture 16 SMP units 512 CUDA cores
786kB L2 cache Note that one CUDA core corresponds to one 5D AMD Stream Processor (VLIW5). Therefore Radeon 5870 has 320 cores with 4-way SIMD capabilities and one SFU. Martin Kruliš

6 NVIDIA Fermi Streaming Multiprocessor 32 CUDA cores
64kB shared memory (or L1 cache) 1024 registers per core 16 load/store units 4 special function units 16 double precision ops per clock 1 instruction decoder All cores are running in lockstep Streaming multiprocessors are designed SM on Fermi, SMX on Kepler, and SMM on Maxwell Martin Kruliš

7 GPU Execution Model Data Parallelism Threading Execution Model
Many data elements are processed concurrently by the same routine GPUs are designed under this particular paradigm Also have limited ways to express task parallelism Threading Execution Model One function (kernel) is executed in many threads Much more lightweight than the CPU threads Threads are grouped into blocks/work groups of the same size Martin Kruliš

8 Instruction Decoder and Warp Scheduler
SIMT Execution Single Instruction Multiple Threads All cores are executing the same instruction Each core has its own set of registers registers Instruction Decoder and Warp Scheduler Martin Kruliš

9 SIMT vs. SIMD Single Instruction Multiple Threads
Width-independent programming model Serial-like code Achieved by hardware with a little help from compiler Allows code divergence Single Instruction Multiple Data Explicitly expose the width of SIMD vector Special instructions Generated by compiler or directly written by programmer Code divergence is usually not supported or tedious Martin Kruliš

10 Simultaneously run on SM cores
Thread-Core Mapping How are threads assigned to SMPs Grid Block Warp Thread The same kernel Simultaneously run on SM cores Assigned to SM Warp is made by subsequent 32 threads of the block according to their thread ID (i.e., threads 0-31, 32-63, …). Warp size is 32 for all compute capabilities. Thread ID is threadIdx.x in one dimension, (threadIdx.x + threadIdx.y*blockDim.x) in two dimensions and threadId.x + threadId.y*blockDim.x + threadId.z*blockDim.x*blockDim.y) in three dimensions. Multiple kernels may run simultaneously on the GPU (since Fermi) and multiple blocks may be assigned simultaneously to an SM (if the registers, the schedulers, and the shared memory can accommodate them). Core GPU Streaming Multiprocessor Martin Kruliš

11 SIMT and Branches Masking Instructions In case of data-driven branches
if-else conditions, while loops, … All branches are traversed, threads mask their execution in invalid branches if (threadId % 2 == 0) { ... we are even ... } else { ... this is odd ... } 1 2 3 4 1 2 3 4 Martin Kruliš

12 GPU Memory Global Memory L2 Cache … Host
Note that details about host memory interconnection are platform specific GPU Device Global Memory L2 Cache L1 Cache Registers Core Host CPU Host Memory GPU Chip SMP ~ 25 GBps PCI Express (16/32 GBps) > 100 GBps Martin Kruliš

13 Host-Device Transfers
PCIe Transfers Much slower than internal GPU data transfers Issued explicitly by host code As a bulk transfer operation Or issued explicitly by the HW/operating system When GPU memory is mapped to host memory space The transfers have significant overhead Overlapping - up to 2 asynchronous transfers may run whilst the GPU is computing Data must be binary safe (no pointers) CUDA introduces unified memory addressing Martin Kruliš

14 Global Memory Global Memory Properties Off-chip, but on the GPU device
High bandwidth and high latency ~ 100 GBps, of clock cycles Operated in transactions Continuous aligned segments of 32 B B Number of transactions caused by global memory access depends on the pattern of the access Certain access patterns are optimized Data are cached in L2 On Fermi also cached in L1 cache Kepler/Maxwell uses L1 for specific purposes Martin Kruliš

15 Shared Memory Memory Shared by the SM Divided into banks
Each bank can be accessed independently Consecutive 32-bit words are in consecutive banks Optionally, 64-bit words division is used (Kepler) Bank conflicts are serialized Except for reading the same address (broadcast) Architecture Mem. size # of banks latency Tesla 16 kB 16 32 bits / 2 cycles Fermi 48 kB 32 Kepler 64 bits / 1 cycle Martin Kruliš

16 Shared Memory Linear Addressing
Each thread in warp access different memory bank No collisions Martin Kruliš

17 Shared Memory Linear Addressing with Stride
Each thread access 2*i-th item 2-way conflicts (2x slowdown) on Fermi and older No collisions on Kepler and newer Due to 64-bits per cycle throughput Martin Kruliš

18 Shared Memory Linear Addressing with Stride
Each thread access 3*i-th item No collisions, since the number of banks is not divisible by the stride Martin Kruliš

19 Registers Registers One register pool per multiprocessor
8-64k of 32-bit registers (depending on architecture) Registers assigned to threads by compiler As fast as the cores (no extra clock cycles) Read-after-write dependency 24 clock cycles Can be hidden if there are enough active warps Hardware scheduler (and compiler) attempts to avoid register bank conflicts whenever possible The programmer have no direct control over them Martin Kruliš

20 Hiding Latency Fast Context Switch When a warp gets stalled
E.g., by data load/store Scheduler switches to the next active warp Martin Kruliš

21 Kepler Architecture Kepler’s Major Improvements
Streaming Processors Next Generation (SMX) 192 cores, 32 SFUs, 32 load/store units 3 cores share a DP unit, 6 cores share LD and SFU Dynamic Parallelism Kernel may spawn child kernels (to depth of 24) Implies the work group context-switch capability Hyper-Q Up to 32 simultaneous GPU-host connections Better throughput if multiple processes/threads use the GPU (concurrent connections are managed in HW) Martin Kruliš

22 Kepler Architecture Martin Kruliš

23 Maxwell Architecture Maxwell’s Major Improvements
Maxwell Symmetric Multiprocessor (SMM) Many internal optimizations, better power efficiency Improved scheduling, increased occupancy Reduced arithmetic instruction latency Larger L2 cache (2MB) Dedicated shared memory (separate L1 cache) Native shared memory atomics Better support for dynamic parallelism Martin Kruliš

24 Maxwell Architecture Martin Kruliš

25 AMD GPU Architecture AMD’s Graphic Core Next (GCN)
Abandoning the VLIW4 architecture 1VLIW x 4 ALU ops => 4 SIMD x 1 ALU op 32 compute units (Radeon HD7970) 4 SIMD units per CU (each processing 16 elements) 10 planned wavefronts per SIMD unit Emphasis on vector processing (instructions, registers, memory, …) OpenCL 1.2, DirectCompute 11.1 and C++ AMP compatibility Martin Kruliš

26 AMD GPU Architecture Martin Kruliš

27 GPU Programming Summary
Data Parallelism SIMT execution model Load balancing needs to be carefully considered Host-device Transfers Data needs to be transferred to the GPU and back Computations should overlap with data transfers There should be sufficient amount of computations Memory Architecture Various types of memories and caches Coalesced loads/stores from/to global memory Banking properties of the shared memory Martin Kruliš

28 Discussion Martin Kruliš

29 OpenCL and GPU Programming
Martin Kruliš Martin Kruliš

30 OpenCL Universal Framework for Parallel Computations
Specification created by Khronos group Multiple implementations exist (AMD, NVIDIA, Mac, …) API for Different Parallel Architectures Multicore CPUs, Manycore GPUs, Xeon Phi, FPGA, … Host runtime handles device detection, data transfers, and code execution Extended Version of C99 for Programming Devices The code is compiled at runtime for selected device Theoretically, we may chose best device for our application dynamically However, we have to consider HW-specific optimizations… Martin Kruliš

31 OpenCL – Hardware Model
Device (CPU die or GPU card) Compute unit (CPU core or GPU SM) Processing element (slot in SSE registers or GPU core) Martin Kruliš

32 Intel Core i7 (4 cores with HT)
OpenCL – Logical Model Logical Layers Platform An implementation of OCL Context Groups devices of selected kind Buffers, programs, and other objects live in a context Device Command Queue Created for a device Controls data transfers, kernel execution and synchronization Intel Core i7 (4 cores with HT) ATI Radeon 5870 (320 cores) Martin Kruliš

33 OpenCL – Client Application
List all platforms std::vector<cl::Platform> platforms; cl_int err = cl::Platform::get(&platforms); if (err != CL_SUCCESS) return 1; cl_context_properties cps[3] = {CL_CONTEXT_PLATFORM, (cl_context_properties)(platforms[0]()), 0}; cl::Context context(CL_DEVICE_TYPE_GPU, cps, NULL, NULL, &err); std::vector<cl::Device> devices = context.getInfo<CL_CONTEXT_DEVICES>(); cl::Buffer buf(context, CL_MEM_READ_ONLY, sizeof(cl_float)*n); cl::Program program(context, cl::Program::Sources(1, std::make_pair(source.c_str(), source.length())) ); err = program.build(devices); cl::Kernel kernel(program, "function_name", &err); err = kernel.setArg(0, buf); Context of all GPUs on the 1st platform List all GPUs Allocate memory on the GPU Create and compile the program from string source Mark function as a kernel Martin Kruliš

34 OpenCL – Client Application
cl::CommandQueue cmdQueue(context, devices[0], 0, &err); cmdQueue.enqueueWriteBuffer(buf, CL_TRUE, 0, sizeof(cl_float)*n, data); cmdQueue.enqueueNDRangeKernel(kernel, cl::NullRange, cl::NDRange(n), cl::NDRange(grp), NULL, NULL); cmdQueue.finish(); cmdQueue.enqueueReadBuffer(buf, CL_TRUE, 0, sizeof(cl_float)*n, data); GPU (in-order) command queue Copy input data to the GPU buffer Execute the kernel and wait for it to finish (not necessary for the following readBuffer operation since the queue is in-order). Copy the results back from the GPU Martin Kruliš

35 OpenCL – Kernels A Kernel Kernel Execution
Written in OpenCL C (extended version of C99) Compiled at runtime for destination platform With (possibly) high degree of optimization Kernel Execution Task Parallelism Multiple kernels are enlisted in command queue(s) and executed concurrently Data Parallelism Multiple instances (threads) are created from single kernel, each operate on its own data elements Note that NVIDIA implementation does not support out-of-order queues at present time. To achieve a true task parallelism or overlap data transfers with kernel execution, multiple queues to the same device must be used. Martin Kruliš

36 OpenCL – Kernel Execution
Data Parallelism Each kernel instance has its own ID A 1-3 dimensional vector of numbers from 0 to N-1 ID identifies the portion of data to be processed Threads form groups (blocks) Threads within one group are executed on one SMP Groups have IDs as well 𝑇 𝐼𝐷 = 𝐺 𝐼𝐷 𝐺 𝑆𝐼𝑍𝐸 + 𝑇 𝐿𝑂𝐶𝐴𝐿_𝐼𝐷 Martin Kruliš

37 OpenCL – Memory Model Types of Memory
private – memory that belongs to one thread (registers) local – memory shared by workgroup (shared memory) global – memory of the device constant – read-only version of global memory More easily cached Martin Kruliš

38 OpenCL – Programming Kernels
Data Types Almost every standard C99 types (int, float, …) Some predefined types: size_t, ptrdiff_t Special type half – 16bit equivalent of float Vector Data Types typeN, where type is std. type and 𝑁∈ 2,4,8,16 E.g. float4, int16 Items are accessed as structure members int4 vec = (int4) (1, 2, 3, 4); int x = vec.x; Special swizzling operations are allowed int4 vec2 = vec.xxyy; vec = vec.wzyx; Martin Kruliš

39 OpenCL – Programming Kernels
Functions Other functions (beside the kernel) can be defined in the program (kernel is just an entry point) Some devices may impose limitations on call stack It is possible to call std. functions like printf() However, they may not work on GPUs There are many built-in functions Thread-related functions (e.g. get_global_id()) Mathematical and geometrical functions Originally designed for graphics Some of them are translated into single instruction Functions for asynchronous memory transfers Martin Kruliš

40 OpenCL – Programming Kernels
Restrictions And Optimization Issues Branching problem (if-else) Workgroup runs in SIMT, thus all branches are followed Use conditional assignment rather than branches For-cycles The compiler attempts to unroll them automatically While-cycles The same problem as branching Vector operations Translated into single instruction if possible (e.g., SSE) The compiler attempts to generate them automatically Different efficiency on different architectures Note that vector operations have virtually no effect on current NVIDIA architectures, but they may be particularly useful on CPUs and AMD GPUs. Martin Kruliš

41 OpenCL – Synchronization
Global Explicit barriers added to the command queue Events and event dependencies Within a Kernel Local barriers (for a work group) Memory fences Atomic operations Common operations (add, sub, xchg, cmpxhg, …) Extended – min, max, and, or, xor Martin Kruliš

42 Example – Matrix Multiplication
__kernel void mul_matrix (__global const float *A, __global const float *B, __global float *C) { int width = get_global_size(0); int x = get_global_id(0); int y = get_global_id(1); float sum = 0; for (int i = 0; i < width; ++i) sum += A[y*width + i] * B[i*width + x]; C[y*width + x] = sum; } Example Martin Kruliš

43 Example – Matrix Multiplication
Exact numbers may vary a little Optimized Solution Workgroup computes block of 16x16 results In each step, appropriate blocks of 16x16 numbers are loaded into local memory and intermediate results are updated × = 16 Example Martin Kruliš

44 OpenCL 2.0 Open CL 2.0 (July 2013) First implementations from Intel and AMD Shared virtual memory For sharing complex data structures with pointers Dynamic parallelism Device kernels can enqueue other kernels Generic address space Functions pointer arguments need not declare their address space Subset of C11 atomics were added Pipes - FIFO data structures, which can be read and written by kernels OpenCL 2.2 specification has been introduced recently. NVIDIA still implements OCL 1.2 only (in stable drivers). Martin Kruliš

45 Discussion Martin Kruliš


Download ppt "GPGPU Architectures Martin Kruliš Martin Kruliš 11. 4. 2017."

Similar presentations


Ads by Google