1. Improving GPU Performance via Large Warps and Two-Level Warp Scheduling
Veynu Narasiman
The University of Texas at Austin
Michael Shebanow
NVIDIA
Chang Joo Lee
Intel
Rustam Miftakhutdinov
The University of Texas at Austin
Onur Mutlu
Carnegie Mellon University
Yale N. Patt
The University of Texas at Austin
MICRO-44
December 6th, 2011
Porto Alegre, Brazil

2. Rise of GPU Computing
GPUs have become a popular platform for general purpose applications
New Programming Models
CUDA
ATI Stream Technology
OpenCL
Order of magnitude speedup over single-threaded CPU
Over the past few years . . .
. . . Allow the programmer to create 1000’s of threads each of which executes the same
Previous work has successfully ported over several applications to the GPU and in some cases

3. How GPUs Exploit Parallelism
Multiple GPU cores (i.e., Streaming Multiprocessors)
Focus on a single GPU core
Exploit parallelism in 2 major ways:
Threads grouped into warps
Single PC per warp
Warps executed in SIMD fashion
Multiple warps concurrently executed
Round-robin scheduling
Helps hide long latencies
The GPU is a massively parallel chip which consists of . . .
Since all threads are executing the same kernel . . .
Warp size for NVIDIA is 32
When a warp executes, all threads within the warp execute the same instruction in parallel across the SIMD resources of the core implying that the SIMD width equals the warp size
The second way . . .
48 warps on NVIDIA hardware
The warps are scheduled in the fetch stage using a . . .
Having so many warps concurrently executing Since when 1 warp is stalled . . .

4. The Problem
Despite these techniques, computational resources can still be underutilized
Two reasons for this:
Branch divergence
Long latency operations

5. Branch Divergence Current PC: B A 1111 Current Active Mask: 1001 1111
Taken
Not Taken
1001
0110 B C D
0110 C D
1111 D
1111 D
Reconverge PC
Active
Mask
Execute PC

6. Long Latency Operations
Core
All Warps Compute
All Warps Compute
Req Warp 0
Memory
System
Req Warp 1
Req Warp 15
time
Round Robin Scheduling, 16 total warps

7. Good
Bad
32 warps, 32 threads per warp, SIMD width = 32, round-robin scheduling

8. Large Warp Microarchitecture (LWM)
Alleviates branch divergence
Fewer, but larger warps
Warp size much greater than SIMD width
Total thread count and SIMD-width stay the same
Dynamically breaks down large warp into sub-warps
Can be executed on existing SIMD pipeline
Rearrange active mask as 2D structure
Number of columns = SIMD width
Search each column for an active thread to create new sub-warp

9. Large Warp Microarchitecture Example
Decode Stage 1 1 1
Sub-warp 1 mask
Sub-warp 0 mask
Sub-warp 2 mask
Sub-warp 1 mask
Sub-warp 0 mask
Sub-warp 0 mask 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

10. More Large Warp Microarchitecture
Divergence stack still used
Handled at the large warp level
How large should we make the warps?
More threads per warp  more potential for sub-warp creation
Too large a warp size can degrade performance
Re-fetch policy for conditional branches
Must wait till last sub-warp finishes
Optimization for unconditional branch instructions
Don’t create multiple sub-warps
Sub-warping always completes in a single cycle

11. Two Level Round Robin Scheduling
Split warps into equal sized fetch groups
Create initial priority among the fetch groups
Round-robin scheduling among warps in same fetch group
When all warps in the highest priority fetch group are stalled
Rotate fetch group priorities
Highest priority fetch group becomes least
Warps arrive at a stalling point at slightly different points in time
Better overlap of computation and memory latency

12. Round Robin vs Two Level Round Robin
Core
All Warps Compute
All Warps Compute
Req Warp 0
Memory
System
Req Warp 1
Req Warp 15
time
Round Robin Scheduling, 16 total warps
Group 0
Group 1
Group 0
Group 1
Core
Compute
Compute
Compute
Compute
Saved Cycles
Req Warp 0
Req Warp 1
Req Warp 7
Memory
System
Req Warp 8
Req Warp 9
Req Warp 15
time
Two Level Round Robin Scheduling, 2 fetch groups, 8 warps each

13. More on Two Level Scheduling
What should the fetch group size be?
Enough warps to keep pipeline busy in the absence of long latency stalls
Too small
Uneven progression of warps in the same fetch group
Destroys data locality among warps
Too large
Reduces benefits of two-level scheduling
More warps stall at the same time
Not just for hiding memory latency
Complex instructions (e.g., sine, cosine, sqrt, etc.)
Two-level scheduling allows warps to arrive at such instructions at slightly different points in time

14. Combining LWM and Two Level Scheduling
4 large warps, 256 threads each
Fetch group size = 1 large warp
Problematic for applications with few long latency stalls
No stalls  no fetch group priority changes
Single large warp starved
Branch re-fetch policy for large warps  bubbles in pipeline
Timeout invoked fetch group priority change
32K instruction timeout period
Alleviates starvation

15. Register File and On Chip Memories
Methodology
Simulate single GPU core with 1024 thread contexts divided into 32 warps each with 32 threads
Scalar Front End
1-wide fetch, decode
4KB single ported I-Cache
Round-robin scheduling
SIMD Back End
In order, 5 stages, 32 parallel SIMD lanes
Register File and On Chip Memories
64KB Register File
128KB, 4-way, D-Cache with 128B line size
128KB, 32-banked private memory
Memory System
Open row, first-come first-serve scheduling
8 banks, 4KB row buffer per bank
100-cycle row hit latency, 300-cycle row conflict latency
32 GB/s memory bandwidth

16. Overall IPC Results
LWM+2Lev improves performance by 19.1% over baseline and by 11.5% over TBC

17. IPC and Computational Resource Utilization

18. Conclusion
For maximum performance, the computational resources on GPUs must be effectively utilized
Branch divergence and long latency operations cause them to be underutilized or unused
We proposed two mechanism to alleviate this
Large Warp Microarchitecture for branch divergence
Two-level scheduling for long latency operations
Improves performance by 19.1% over traditional GPU cores
Increases scope of applications that can run efficiently on a GPU
Questions