1. Prepared 8/8/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron.

2. The Problem: how do you do global communication? Finish a kernel and start a new one All writes from all threads complete before a kernel finishes Would need to decompose kernels into before and after parts CUDA Threads and Atomics – Slide 2 step1 >>(…); // The system ensures that all // writes from step 1 complete step2 >>(…);

3. Alternately write to a predefined memory location Race condition! Updates can be lost What is the value of a in thread 0? In thread 1917? CUDA Threads and Atomics – Slide 3 threadID: 0threadID: 1917 // vector[0] was equal to zero vector[0] += 5; … a = vector[0]; vector[0] += 1; … a = vector[0];

4. Thread 0 could have finished execution before 1917 started Or the other way around Or both are executing at the same time Answer: not defined by the programming model; can be arbitrary CUDA Threads and Atomics – Slide 4 threadID: 0threadID: 1917 // vector[0] was equal to zero vector[0] += 5; … a = vector[0]; vector[0] += 1; … a = vector[0];

5. CUDA provides atomic operations to deal with this problem An atomic operation guarantees that only a single thread has access to a piece of memory while an operation completes The name atomic comes from the fact that it is uninterruptable No dropped data, but ordering is still arbitrary CUDA Threads and Atomics – Slide 5

6. CUDA provides atomic operations to deal with this problem Requires hardware with compute capability 1.1 and above Different types of atomic instructions Addition/subtraction: atomicAdd, atomicSub Minimum/maximum: atomicMin, atomicMax Conditional increment/decrement: atomicInc, atomicDec Exchange/compare-and-swap: atomicExch, atomicCAS More types in fermi: atomicAnd, atomicOr, atomicXor CUDA Threads and Atomics – Slide 6

7. CUDA Threads and Atomics – Slide 7 // Determine frequency of colors in a picture // colors have already been converted into integers // Each thread looks at one pixel and increments // a counter atomically __global__ void histogram(int* color, int* buckets) { int i = threadIdx.x + blockDim.x * blockIdx.x; int c = colors[i]; atomicAdd(&buckets[c], 1); } atomicAdd returns the previous value at a certain address Useful for grabbing variable amounts of data from the list

8. CUDA Threads and Atomics – Slide 8 // For algorithms where the amount of work per item // is highly non-uniform, it often makes sense for // to continuously grab work from a queue __global__ void workq(int* work_q, int* q_counter, int* output, int queue_max) { int i = threadIdx.x + blockDim.x * blockIdx.x; int q_index = atomicInc(q_counter, queue_max); int result = do_work(work_q[q_index]); output[i] = result; }

9. if compare equals old value stored at address then val is stored instead in either case, routine returns the value of old seems a bizarre routine at first sight, but can be very useful for atomic locks CUDA Threads and Atomics – Slide 9 int atomicCAS(int* address, int compare, int val)

10. Most general type of atomic Can emulate all others with CAS CUDA Threads and Atomics – Slide 10 int atomicCAS(int* address, int oldval, int val) { int old_reg_val = *address; if (old_reg_val == compare) *address = val; return old_reg_val; }

11. Atomics are slower than normal load/store Most of these are associative operations on signed/unsigned integers: quite fast for data in shared memory slower for data in device memory You can have the whole machine queuing on a single location in memory Atomics unavailable on G80! CUDA Threads and Atomics – Slide 11

12. CUDA Threads and Atomics – Slide 12 // If you require the maximum across all threads // in a grid, you could do it with a single global // maximum value, but it will be VERY slow __global__ void global_max(int* values, int* gl_max) { int i = threadIdx.x + blockDim.x * blockIdx.x; int val = values[i]; atomicMax(gl_max,val); }

13. CUDA Threads and Atomics – Slide 13 // introduce intermediate maximum results, so that // most threads do not try to update the global max __global__ void global_max(int* values, int* max, int *regional_maxes, int num_regions) { int i = threadIdx.x + blockDim.x * blockIdx.x; int val = values[i]; int region = i % num_regions; if (atomicMax(&reg_max[region],val) < val) { atomicMax(max,val); }

14. Single value causes serial bottleneck Create hierarchy of values for more parallelism Performance will still be slow, so use judiciously CUDA Threads and Atomics – Slide 14

15. Can’t use normal load/store for inter-thread communication because of race conditions Use atomic instructions for sparse and/or unpredictable global communication Decompose data (very limited use of single global sum/max/min/etc.) for more parallelism CUDA Threads and Atomics – Slide 15

16. How a streaming multiprocessor (SM) executes threads Overview of how a streaming multiprocessor works SIMT Execution Divergence CUDA Threads and Atomics – Slide 16

17. Hardware schedules thread blocks onto available SMs No guarantee of ordering among thread blocks Hardware will schedule thread blocks as soon as a previous thread block finished CUDA Threads and Atomics – Slide 17 Thread Block 5 Thread Block 27 Thread Block 61 Streaming Multiprocessor Thread Block 2001

18. A warp = 32 threads launched together Usually execute together as well CUDA Threads and Atomics – Slide 18 ALU Control ALU Control ALU Control ALU Control ALU Control ALU Control ALU Control ALU

19. Each thread block is mapped to one or more warps The hardware schedules each warp independently CUDA Threads and Atomics – Slide 19 Thread Block N (128 threads) TB N W1 TB N W2 TB N W3 TB N W4

20. SM implements zero-overhead warp scheduling At any time, only one of the warps is executed by SM* Warps whose next instruction has its inputs ready for consumption are eligible for execution Eligible warps are selected for execution on a prioritized scheduling policy All threads in a warp execute the same instruction when selected CUDA Threads and Atomics – Slide 20

21. Threads are executed in warps of 32, with all threads in the warp executing the same instruction at the same time What happens if you have the following code? CUDA Threads and Atomics – Slide 21 if (foo(threadIdx.x)) { do_A(); } else { do_B(); }

22. This is called warp divergence – CUDA will generate correct code to handle this, but to understand the performance you need to understand what CUDA does with it CUDA Threads and Atomics – Slide 22 if (foo(threadIdx.x)) { do_A(); } else { do_B(); }

23. CUDA Threads and Atomics – Slide 23 From Fung et al. MICRO ’07 Branch Path A Path B Branch Path A Path B

24. Nested branches are handled as well CUDA Threads and Atomics – Slide 24 if (foo(threadIdx.x)) { if (bar(threadIdx.x)) do_A(); else do_B(); } else do_C();

25. CUDA Threads and Atomics – Slide 25 Branch Path A Path C Branch Path B

26. You don’t have to worry about divergence for correctness Mostly true, except corner cases (for example intra-warp locks) You might have to think about it for performance Depends on your branch conditions CUDA Threads and Atomics – Slide 26

27. One solution: NVIDIA GPUs have predicated instructions which are carried out only if a logical flag is true. In the previous example, all threads compute the logical predicate and two predicated instructions CUDA Threads and Atomics – Slide 27 p = (foo(threadIdx.x)); p: do_A(); !p: do_B();

28. Performance drops off with the degree of divergence CUDA Threads and Atomics – Slide 28 switch (threadIdx.x % N) { case 0:... case 1:... }

29. Performance drops off with the degree of divergence In worst case, effectively lose a factor of 32 in performance if one thread needs expensive branch, while rest do nothing Another example: processing a long list of elements where, depending on run-time values, a few require very expensive processing GPU implementation: first process list to build two sub-lists of “simple” and “expensive” elements then process two sub-lists separately CUDA Threads and Atomics – Slide 29

30. Already introduced __syncthreads(); which forms a barrier – all threads wait until every one has reached this point. When writing conditional code, must be careful to make sure that all threads do reach the __syncthreads(); Otherwise, can end up in deadlock CUDA Threads and Atomics – Slide 30

31. Fermi supports some new synchronisation instructions which are similar to __syncthreads() but have extra capabilities: int __syncthreads_count(predicate) counts how many predicates are true int __syncthreads_and(predicate) returns non-zero (true) if all predicates are true int __syncthreads_or(predicate) returns non-zero (true) if any predicate is true CUDA Threads and Atomics – Slide 31

32. There are similar warp voting instructions which operate at the level of a warp: int __all(predicate) returns non-zero (true) if all predicates in warp are true int __any(predicate) returns non-zero (true) if any predicate is true unsigned int __ballot(predicate) sets n th bit based on n th predicate CUDA Threads and Atomics – Slide 32

33. Use very judiciously Always include a max_iter in your spinloop! Decompose your data and your locks CUDA Threads and Atomics – Slide 33

34. Problem: when a thread writes data to device memory the order of completion is not guaranteed, so global writes may not have completed by the time the lock is unlocked CUDA Threads and Atomics – Slide 34 // global variable: 0 unlocked, 1 locked __device__ int lock=0; __global__ void kernel(...) {... // set lock do {} while(atomicCAS(&lock,0,1));... // free lock lock = 0; }

35. CUDA Threads and Atomics – Slide 35 // global variable: 0 unlocked, 1 locked __device__ int lock=0; __global__ void kernel(...) {... // set lock do {} while(atomicCAS(&lock,0,1));... // free lock __threadfence(); // wait for writes to finish lock = 0; }

36. __threadfence_block(); wait until all global and shared memory writes are visible to all threads in block __threadfence(); wait until all global and shared memory writes are visible to all threads in block (or all threads, for global data) CUDA Threads and Atomics – Slide 36

37. lots of esoteric capabilities – don’t worry about most of them essential to understand warp divergence – can have a very big impact on performance __syncthreads(); is vital the rest can be ignored until you have a critical need – then read the documentation carefully and look for examples in the SDK CUDA Threads and Atomics – Slide 37

38. Based on original material from Oxford University: Mike Giles Stanford University Jared Hoberock, David Tarjan Revision history: last updated 8/8/2011. CUDA Threads and Atomics – Slide 38