1. CUDA Programming continued ITCS 4145/5145 Nov 24, 2010 © Barry Wilkinson Revised

2. 2 Timing GPU Execution Can use CUDA “events” – create two events and compute the time between them: cudaEvent_t start, stop; float elapsedTime; cudaEventCreate(&start); // create event objects cudaEventCreate(&stop); cudaEventRecord(start, 0); // Record start event. cudaEventRecord(stop, 0); // record end event cudaEventSynchronize(stop); // wait work preceding to complete cudaEventRecord(stop,0) cudaEventElapsedTime(&elapsedTime, start, stop); //compute elapsed time between events cudaEventDestroy(start);//destroy start event cudaEventDestroy(stop);); //destroy stop event

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8 Host Synchronization Kernels Control returned to CPU immediately (asynchronous, non-blocking) Kernel starts after all previous CUDA calls completed cudaMemcpy Returns after copy complete (synchronous) Copy starts after all previous CUDA calls completed

9. 9 CUDA Synchronization Routines Host cudaThreadSynchronize() Blocks until all previous CUDA calls complete GPU void __syncthreads() Synchronizes all threads in a block Barrier – no thread can pass until all threads in block reach it. All threads must reach __syncthread in thread block.

10. 10 GPU Atomic Operations Performs a read-modify-write atomic operation on one word residing in global or shared memory. Associative operations on signed/unsigned integers, add, sub, min, max, and, or, xor, increment, decrement, exchange, compare and swap. Requires GPU with compute capability 1.1+ (Shared memory operations and 64-bit words require higher capability) coit-grid06 Tesla C2050 has compute capability 2.0 See http://www.nvidia.com/object/cuda_gpus.html for GPU compute capabilities

11. 11 int atomicAdd(int* address, int val); reads old located at address address in global or shared memory, computes (old + val), and stores result back to memory at same address. These three operations (read, compute, and write) are performed in one atomic transaction.* Function returns old. Atomic Operation Example * Once stated, it continues to completion without being able to be interrupted by other processors. Other processors cannot read or write to memory location once atomic operation starts. Mechanism implemented in hardware.

12. 12 Other operations int atomicSub(int* address, int val); int atomicExch(int* address, int val); int atomicMin(int* address, int val); int atomicMax(int* address, int val); unsigned int atomicInc(unsigned int* address, unsigned int val); unsigned int atomicDec(unsigned int* address, unsigned int val); int atomicCAS(int* address, int compare, int val); //compare and swap int atomicAnd(int* address, int val); int atomicOr(int* address, int val); int atomicXor(int* address, int val); Source: NVIDIA CUDA C Programming Guide, version 3.2, 11/9/2010

13. 13 int atomicCAS(int* address, int compare, int val); reads the word old located at address address in global or shared memory, and compares old with compare. If they are the same, it set old to val (stores val at address address), i.e.: if (old == compare) old = val; // else old = old The three operations (read, compute, and write) are performed in one atomic transaction. The function returns the original value of old. Also unsigned and unsigned long long int versions. Compare and Swap (also called compare and exchange)

14. 14 __device__ int lock=0; // unlocked __global__ void kernel(...) {... do {} while(atomicCAS(&lock,0,1)); // if lock = 0 set to1 // and continue... // critical section lock = 0; // free lock } Coding Critical Sections with Locks

15. 15 Memory Fences Threads may see the effects of a series of writes to memory executed by another thread in different orders. To enforce ordering: void __threadfence_block(); waits until all global and shared memory accesses made by the calling thread prior to __threadfence_block() are visible to all threads in the thread block. Other routines: void __threadfence(); void __threadfence_system();

16. 16 Writes to device memory not guaranteed in any order, so global writes may not have completed by the time the lock is unlocked __global__ void kernel(...) {... do {} while(atomicCAS(&lock,0,1));...// criticial section __threadfence(); // wait for writes to finish lock = 0; } Critical sections with memory operations

17. 17 Error reporting All CUDA calls (except kernel launches) return error code of type cudaError_t cudaError_t cudaGetLastError(void) Returns code for the last error Can be used to get error from kernel execution. Char* cudaGetErroprString(cudaError_t code) Returns a null-terminated character string describing error Example print(“%s\n”,cudaGetErrorString(cudaGetLastError());

18. Questions