Some key aspects of NVIDIA GPUs and CUDA

Silicon Usage

Performance and Bandwidth

PCIe Connected Separate memory space on CPU (host) and GPU (device) Must malloc space on GPU and copy data from CPU memory to GPU memory Need means to copy executable to GPU and start it (kernel launch)

Hardware Architecture

CUDA Streaming Multiprocessor 32 cores assigned to 32 threads (a warp) Limited number of SFUs In-order execution Latency masked by swamping between warps of threads

Warp Processing (hide latency)

Memory Model

Blocks and Grids