1. GPUs: Overview of Architecture and Programming Options Lee Barford firstname dot lastname at gmail dot com

2. Outline Why parallel computing is now important What GPUs are and what they provide Overview of GPU architecture Enough to orient the discussion of programming them Future changes Three “languages” for programming GPUs Those we’re not doing include CUDAFortran, Python CUDA & CL bindings, WebCL

3. 3 Graph from UC Berkeley ParLab Serial App Performance Exponentially growing gap

4. Graphics Processor (GPU) as Parallel Accelerator Commodity priced, massively parallel floating point Claimed performance on various problems 50-2500x CPU running serial code 4 Graph from http://drdobbs.com/high-performance-computing/231500166

5. The GPU as a Co-Processor to the CPU: The physical and logical connections Main memory chipset GPU memory PCIe Slow Control actions & code (kernels) to run I/Os: Video Ethernet USB hub Firewire … CPU GPU Running GPU code is like requesting asynchronous I/O

6. 0.5-3 years from now: Fusion of CPU and GPU CPU Main memory I/O subsystem Multiple cores GPU Running GPU code will be like pending method pointers for future execution. (Like C++11, TBB, TPL, PPL). Hardware task scheduler

7. Programming Tomorrow’s CPU will be Like Programming Today’s GPU GPUs that compute will come “for free” with computers Slow step of moving data to/from GPU will be eliminated Hardware task scheduler for both CPU and GPU will Almost eliminate OS & I/O overhead for invoking GPU kernels Also almost eliminate OS overhead for invoking parallel tasks on CPU AMD laptop chip available now (but no boards/systems) NVIDIA GPU+ARM chip available now for battery operated devices Both promise desktop chips in next year or two Programming models will probably evolve from what we’ll cover Course will use current, PCIe-based GPUs We will be dealing with overheads that will pass away over next few years

8. CUDA (NVIDIA) GPU Compute Architecture: Many Simple, Floating-Point Cores

9. 32 cores (Streaming Multiprocessor) share: Instruction stream Registers Execute same program (kernel) SPMD: ~ [Same place in same kernel at the same time] Act as 100-1000’s more cores by switching context instead of waiting for memory 1000’s of virtual cores executing same lines of code together, but Sharing limited resources Cores organized into groups

10. GPU has multiple SMs SMs run in parallel Do not need to be executing same location in the same program at the same time In aggregate, many 1000’s of parallel copies of same kernel running simultaneously Total of up to 1Tflop/s at peak CENTRAL SOFTWARE ISSUE: How to generate and control this much parallelism

11. GPUs: Programming Options Libraries: called from CPU code. Write no GPU code. Examples: Image/video processing, dense & sparse matrix, FFT, random numbers Generic programming for GPU Thrust Like C++ Standard Template Library Specialize & use built-in data structures and algorithms NVIDIA GPUs only Programming the GPU directly CUDA C/C++, OpenCL, WebCL, CUDA Fortran, various Python libraries Write code that runs on GPU (kernels) Write CPU code that directly controls and coordinates –Data movement between CPU memory and GPU memory –Startup of kernels on GPU –CPU processing of results from GPU when they become available

12. CUDA C/C++ vs OpenCL CUDA C/C++ Proprietary (NVIDIA) Code runs on NVIDIA GPUs Reportedly 10-50% faster than OpenCL Compiles at build time to binary code for particular targeted hardware Specific NVIDIA hardware architecture versions No compiler available at run time OpenCL Open standard (Khronos) Code runs on NVIDIA & AMD GPUs, x86 multicore, FPGAs (academic research) at the same time Compiles at build time to intermediate form that is compiled at run time for the hardware that is present Compiler is available at run time Can execute downloaded or dynamically generated source code

13. The Three Programming Environments We’ll Cover OpenCL : Write once, run many Supports heterogeneous parallel machines (fusion) Tool chains good enough for research IMHO, will eventually replace CUDA C/C++ OpenCL : Write once, run many Supports heterogeneous parallel machines (fusion) Tool chains good enough for research IMHO, will eventually replace CUDA C/C++ CUDA C/C++: Very efficient code Lots of fussy detail to get that efficiency Robust tool chains for Linux, Windows, MacOS Specific to NVIDIA CUDA C/C++: Very efficient code Lots of fussy detail to get that efficiency Robust tool chains for Linux, Windows, MacOS Specific to NVIDIA Thrust: Easy to write Algorithms provided among the fastest (e.g., sort) NVIDIA GPUs only Thrust: Easy to write Algorithms provided among the fastest (e.g., sort) NVIDIA GPUs only

14. Class Project Idea Accurate edge finding in a 1D signal Journal paper published on multicore version Student project last year doing Thrust implementation Project: Do CUDA version + performance tests Paper combining previous student’s work with above: 60% probability of getting accepted in a particular IEEE conference 3 co-authors, including previous student & Lee Extended abstract due: Nov 6 Class project due during finals, same as everyone else Camera ready paper due: March 4 See or email me in the next week or two if interested

15. Questions