1. Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow Wilson W. L. Fung Ivan Sham George Yuan Tor M. Aamodt Electrical and Computer Engineering University of British Columbia Micro-40 Dec 5, 2007

2. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 2 Motivation = GPU: A massively parallel architecture  SIMD pipeline: Most computation out of least silicon/energy Goal: Apply GPU to non-graphics computing  Many challenges  This talk: Hardware Mechanism for Efficient Control Flow

3. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 3 Programming Model Modern graphics pipeline CUDA-like programming model  Hide SIMD pipeline from programmer  Single-Program-Multiple-Data (SPMD)  Programmer expresses parallelism using threads  ~Stream processing Vertex Shader Pixel Shader OpenGL/ DirectX

4. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 4 Programming Model Warp = Threads grouped into a SIMD instruction From Oxford Dictionary:  Warp: In the textile industry, the term “warp” refers to “the threads stretched lengthwise in a loom to be crossed by the weft”.

5. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 5 The Problem: Control flow GPU uses SIMD pipeline to save area on control logic.  Group scalar threads into warps Branch divergence occurs when threads inside warps branches to different execution paths. Branch Path A Path B Branch Path A Path B 50.5% performance loss with SIMD width = 16

6. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 6 Dynamic Warp Formation Consider multiple warps Branch Path A Path B Opportunity? Branch Path A 20.7% Speedup with 4.7% Area Increase

7. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 7 Outline Introduction Baseline Architecture Branch Divergence Dynamic Warp Formation and Scheduling Experimental Result Related Work Conclusion

8. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 8 Baseline Architecture Shader Core Interconnection Network Memory Controller GDDR3 Memory Controller GDDR3 Memory Controller GDDR3 Shader Core Shader Core CPU spawn done GPU CPU Time CPU spawn GPU

9. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 9 SIMD Execution of Scalar Threads All threads run the same kernel Warp = Threads grouped into a SIMD instruction Thread Warp 3 Thread Warp 8 Thread Warp 7 Thread Warp Scalar Thread W Scalar Thread X Scalar Thread Y Scalar Thread Z Common PC SIMD Pipeline

10. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 10 Latency Hiding via Fine Grain Multithreading Interleave warp execution to hide latencies Register values of all threads stays in register file Need 100~1000 threads  Graphics has millions of pixels

11. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 11 Thread Warp Common PC SPMD Execution on SIMD Hardware: The Branch Divergence Problem Thread 2 Thread 3 Thread 4 Thread 1 B CD E F A G

12. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 12 -G1111 TOS B CD E F A G Baseline: PDOM Thread Warp Common PC Thread 2 Thread 3 Thread 4 Thread 1 B/1111 C/1001D/0110 E/1111 A/1111 G/1111 -A1111 TOS ED0110 EC1001 TOS -E1111 ED0110 TOS -E1111 ADGA Time CBE -B1111 TOS -E1111 TOS Reconv. PC Next PCActive Mask Stack ED0110 EE1001 TOS -E1111

13. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 13 Dynamic Warp Formation: Key Idea Idea: Form new warp at divergence  Enough threads branching to each path to create full new warps

14. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 14 Dynamic Warp Formation: Example AABBGGAACCDDEEFF Time AABBGGAA CD EE F A x/1111 y/1111 B x/1110 y/0011 C x/1000 y/0010 D x/0110 y/0001 F x/0001 y/1100 E x/1110 y/0011 G x/1111 y/1111 A new warp created from scalar threads of both Warp x and y executing at Basic Block D D Execution of Warp x at Basic Block A Execution of Warp y at Basic Block A Legend AA Baseline Dynamic Warp Formation

15. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 15 A 5678 A 1234 Dynamic Warp Formation: Hardware Implementation 578 6 B C 1011 0100 B 23 0110B 0 B 5238 B 0010B 2 7 1 3 4 2 B C 0110 1001 C 1 C 1 4 C 6 1101C 1 No Lane Conflict A: BEQ R2, B C: … X 1 2 3 4 Y 5 6 7 8 X 1 2 3 4 X 1 2 3 4 X 1 2 3 4 X 1 2 3 4 Y 5 6 7 8 Y 5 6 7 8 Y 5 6 7 8 Y 5 6 7 8 Z 5 2 3 8 Z 5 2 3 8 Z 5 2 3 8

16. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 16 Methodology Created new cycle-accurate simulator from SimpleScalar (version 3.0d) Selected benchmarks from SPEC CPU2006, SPLASH2, CUDA Demo  Manually parallelized  Similar programming model to CUDA

17. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 17 Experimental Results 0 16 32 48 64 80 96 112 128 hmmerlbmBlackBitonicFFTLUMatrixHM IPC Baseline: PDOM Dynamic Warp Formation MIMD

18. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 18 Dynamic Warp Scheduling Lane Conflict Ignored (~5% difference)

19. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 19 Area Estimation CACTI 4.2 (90nm process) Size of scheduler = 2.471mm 2 8 x 2.471mm 2 + 2.628mm 2 = 22.39mm 2  4.7% of Geforce 8800GTX (~480mm 2 )

20. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 20 Related Works Predication  Convert control dependency into data dependency Lorie and Strong  JOIN and ELSE instruction at the beginning of divergence Cervini  Abstract/software proposal for “regrouping”  SMT processor Liquid SIMD (Clark et al.)  Form SIMD instructions from scalar instructions Conditional Routing (Kapasi)  Code transform into multiple kernels to eliminate branches

21. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 21 Conclusion Branch divergence can significantly degrade a GPU’s performance.  50.5% performance loss with SIMD width = 16 Dynamic Warp Formation & Scheduling  20.7% on average better than reconvergence  4.7% area cost Future Work  Warp scheduling – Area and Performance Tradeoff

22. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 22 Thank You. Questions?

23. Wilson Fung, Ivan Sham, George Yuan, Tor Aamodt Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow 23 Shared Memory Banked local memory accessible by all threads within a shader core (a block) Idea: Break Ld/St into 2 micro-code:  Address Calculation  Memory Access After address calculation, use bit vector to track bank access just like lane conflict in the scheduler