Basic CUDA Programming Shin-Kai Chen skchen@twins.ee.nctu.edu.tw VLSI Signal Processing Laboratory Department of Electronics Engineering National Chiao Tung University

What will you learn in this lab? Concept of multicore accelerator Multithreaded/multicore programming Memory optimization

Slides Mostly from Prof. Wen-Mei Hwu of UIUC http://courses.ece.uiuc.edu/ece498/al/Syllabus.html

CUDA – Hardware? Software?

Host-Device Architecture CPU (host) GPU w/ local DRAM (device)

G80 CUDA mode – A Device Example Load/store Global Memory Thread Execution Manager Input Assembler Host Texture Parallel Data Cache

Functional Units in G80 Streaming Multiprocessor (SM) 1 instruction decoder ( 1 instruction / 4 cycle ) 8 streaming processor (SP) Shared memory SM 0 SM 1 SP Shared Memory MT IU SP Shared Memory MT IU t0 t1 t2 … tm t0 t1 t2 … tm Blocks Blocks

Setup CUDA for Windows

CUDA Environment Setup Get GPU that support CUDA http://www.nvidia.com/object/cuda_learn_products.html Download CUDA http://www.nvidia.com/object/cuda_get.html CUDA driver CUDA toolkit CUDA SDK (optional) Install CUDA Test CUDA Device Query

Setup CUDA for Visual Studio From scratch http://forums.nvidia.com/index.php?showtopic=30273 CUDA VS Wizard http://sourceforge.net/projects/cudavswizard/ Modified from existing project

Lab1: First CUDA Program

CUDA Computing Model

Data Manipulation between Host and Device cudaError_t cudaMalloc( void** devPtr, size_t count ) Allocates count bytes of linear memory on the device and return in *devPtr as a pointer to the allocated memory cudaError_t cudaMemcpy( void* dst, const void* src, size_t count, enum cudaMemcpyKind kind) Copies count bytes from memory area pointed to by src to the memory area pointed to by dst kind indicates the type of memory transfer cudaMemcpyHostToHost cudaMemcpyHostToDevice cudaMemcpyDeviceToHost cudaMemcpyDeviceToDevice cudaError_t cudaFree( void* devPtr ) Frees the memory space pointed to by devPtr

Example Functionality: Given an integer array A holding 8192 elements For each element in array A, calculate A[i]256 and leave the result in B[i]

Now, go and finish your first CUDA program !!!

Download http://twins.ee.nctu.edu.tw/~skchen/lab1.zip Open project with Visual C++ 2008 ( lab1/cuda_lab/cuda_lab.vcproj ) main.cu Random input generation, output validation, result reporting device.cu Lunch GPU kernel, GPU kernel code parameter.h Fill in appropriate APIs GPU_kernel() in device.cu

Lab2: Make the Parallel Code Faster

Parallel Processing in CUDA Parallel code can be partitioned into blocks and threads cuda_kernel<<<nBlk, nTid>>>(…) Multiple tasks will be initialized, each with different block id and thread id The tasks are dynamically scheduled Tasks within the same block will be scheduled on the same stream multiprocessor Each task take care of single data partition according to its block id and thread id

Locate Data Partition by Built-in Variables gridDim x, y blockIdx blockDim x, y, z threadIdx

Data Partition for Previous Example When processing 64 integer data: cuda_kernel<<<2, 2>>>(…) int total_task = gridDim.x * blockDim.x ; int task_sn = blockIdx.x * blockDim.x + threadIdx.x ; int length = SIZE / total_task ; int head = task_sn * length ;

Processing Single Data Partition

Parallelize Your Program !!!

Partition kernel into threads Increase nTid from 1 to 512 Keep nBlk = 1 Group threads into blocks Adjust nBlk and see if it helps Maintain total number of threads below 512, e.g. nBlk * nTid < 512

Lab3: Resolve Memory Contention

Parallel Memory Architecture Memory is divided into banks to achieve high bandwidth Each bank can service one address per cycle Successive 32-bit words are assigned to successive banks

Lab2 Review When processing 64 integer data: cuda_kernel<<<1, 4>>>(…)

How about Interleave Accessing? When processing 64 integer data: cuda_kernel<<<1, 4>>>(…)

Implementation of Interleave Accessing cuda_kernel<<<1, 4>>>(…) head = task_sn stripe = total_task

Improve Your Program !!!

Modify original kernel code in interleaving manner cuda_kernel() in device.cu Adjusting nBlk and nTid as in Lab2 and examine the effect Maintain total number of threads below 512, e.g. nBlk * nTid < 512

Thank You http://twins.ee.nctu.edu.tw/~skchen/lab3.zip Final project issue Subject: Porting & optimizing any algorithm on any multi-core Demo: 1 week after final exam @ ED412 Group: 1 ~ 2 person per group * Group member & demo time should be registered after final exam @ ED412