SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Sponsors: National Science Foundation, LogicBlox Inc., and NVIDIA Kernel Weaver: Automatically Fusing Database Primitives for Efficient GPU Computation Haicheng Wu 1, Gregory Diamos 2, Srihari Cadambi 3, Sudhakar Yalamanchili 1 1 Georgia Institute of Technology 2 NVIDIA Research 3 NEC Laboratories America

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Data Warehousing Applications on GPUs 2 The Opportunity Significant potential data parallelism If data fits in GPU memory, 2x—27x speedup has been shown 1 The Challenge Need to process 1-50 TBs of data 2 15–90% of the total time * spent in moving data between CPU and GPU * Fine grained computation 1 B. He, M. Lu, K. Yang, R. Fang, N. K. Govindaraju, Q. Luo, and P. V. Sander. Relational query co-processing on graphics processors. In TODS, Independent Oracle Users Group. A New Dimension to Data Warehousing: 2011 IOUG Data Warehousing Survey.

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Relational Algebra (RA) Operators——Select RA operators are the building blocks of DB applications Set Intersection Set Union Set Difference Cross Product Join Select Project KeyValue 3True, a 3False, b 4True, a Example: Select [Key == 3] KeyValue 3True, a 3False, b 4True, a 3

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Relational Algebra (RA) Operators ——Join RA are building blocks of DB applications Set Intersection Set Union Set Difference Cross Product Join Select Project KeyValue 3a 3b 4a KeyValue 3c 4d 5e Example: Join KeyValue 3a,c 3b,c 4a,d New Key = Key(A) ∩ Key(B) New Vallue = Value(A) U Value(B) A B JOIN (A, B) 4

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Data Movement in Kernel Execution 5 ~250GB/s ① Input ② Execute ③ Result M N T

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Memory Hierarchy Bottleneck

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion GPU MEM GPU Core A1 A2 A3 Temp A1 A2 A3 Temp Result Before Fusion GPU MEM GPU Core A1 A2 A3 A1 A2 A3 Result After Fusion Temp Kernel AKernel B Fused Kernel A&B Kernel A A1A2 A3 Kernel B Result Temp A1A2A3 Fused Kernel A, B Result 7

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Major Benefits Reduce Data Footprint Reduction in accesses to global memory Access to common data across kernels improves temporal locality Reduction in PCIe transfers Expand optimization scope of the compiler Data re-use Increase textual scope of optimizers 8 Kernel A A1A2 A3 Kernel B Result Temp A1A2A3 Fused Kernel A, B Result

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Outline Introduction Current RA implementation Kernel Fusion Kernel fusion criteria Greedy heuristic to choose candidates Weaving and fusion Experiment Results Conclusion 9

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Red Fox Compilation Flow 10 RA-to-PTX (nvcc + RA-Lib) Runtime LogicBlox Front-End Language Front-End Translation Layer Back-End Datalog Queries Query PlanPTX/Binary Kernel Kernel Weaver Kernel Weaver – CUDA source to source transformation to apply kernel fusion PTX – Parallel Thread Execution RA Primitives Library

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Example of SELECT * G. Diamos, H. Wu, J. Wang, A. Lele, and S. Yalamanchili. Relational Algorithms for Multi-Bulk-Synchronous Processors. In PPoPP, RA Implementation —— Multi-Stage Algorithms 11 All primitives have the same three stages * Each stage normally maps to 1 CUDA kernel

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion – Three Steps 1.Opportunity: Find candidates meeting fusion criteria. 2.Feasibility: Choose kernels to fuse according to available resources. 3.Fusion: Kernel fusion. 12

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion Criteria (1) 13 Compatible kernel configurations (CTA & thread dimensions) Implementations of RA primitives are parametric Empirically choose configurations after fusion M1 N1 M2 N2 T1 T2 M N T Kernel A Kernel B Fused Kernel A & B 13

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Dependence Restriction Thread dependence Kernel Fusion Criteria (2) Kernel A Kernel B Input data have 2 attributes Operations of each thread are independent Use registers to communicate 14 Kernel A Kernel B

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Dependence Restriction Thread dependence CTA (Thread Block) dependence Kernel Fusion Criteria (2) 15 Kernel A Kernel B 15

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion Criteria-CTA Dependence 16 Threads in the same CTA have dependence No dependence between CTAs Can be fused After fusion Use Shared MEM to communicate Synchronization is needed Example of 2 back-to-back JOINs

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Dependence Restriction Thread dependence CTA (Thread Block) dependence Kernel dependence Kernel Fusion Criteria (2) 17 Can be fused Kernel A Kernel B 17

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion Criteria - Candidates for Fusion Only exhibit thread or CTA dependence Bounded by operators with kernel dependence 18

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Choosing Operators to Fuse 19 Dependence Graph 1. Topo Sort 2. Incrementally add operators 3. Stop When the Estimated Usage is Larger than Budget Kernel fusion will increase resource usage, e.g., registers Greedy heuristic to choose

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Weaving and Fusion 20 Interweaving and Fusing individual stages (CUDA kernels) Use registers or shared memory to store temporary result

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Fusing Thread Dependent Only Operators 21 Example of fusing 2 SELECTs  Unary operators only  No Synchronization required  Register-based communication Select

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Gather Partition Compute Fusing CTA and Thread Dependent Operators 22  Partition multiple inputs  Synchronization necessary  Communication via shared memory Example Pattern

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Experimental Environment CPU2 quad-core Xeon 2.27GHz Memory48 GB GPU1 Tesla C2070 (6GB GDDR5 memory) OSUbuntu Server GCC4.4.3 NVCC4.0 Use micro-benchmarks derived from TPC-H Measure memory allocation, memory access demand, effect of optimization scope, and PCIe traffic Full queries from TPC-H 23

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY TPC-H Benchmark Suites 24 A popular decision making benchmark suite Micro-benchmarks are common patterns from TPC-H Baseline: directly using primitive implementation without fusion Optimized: fusing all primitives of each pattern

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Fused vs. Not Fused Small Inputs-PCIe excluded 25 Average 2.89x speedup Small inputs (64MB-1GB) fitting the GPU memory

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Small Inputs-Analysis 26 Memory Allocation Compiler Optimization (Speedup of O3) Memory Access Reduction

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Conclusions Kernel Fusion can reduce data transfer and speeds up the computation for Data Warehousing Apps. Definition of basic dependences and general criteria for kernel fusion applicable across multiple application domains Quantification of the impact of kernel fusion on different levels of the CPU-GPU memory hierarchy for a range of RA operators. Proposes and demonstrates the utility of compile-time data movement optimizations based on kernel fusion 27

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Thank You Questions? 28

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Backup Slides 29

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Kernel Fusion- A Data Movement Optimization 30 Increase the granularity of kernel computation Reduce data movement throughout the hierarchy Inspired by loop fusion Compile-time automation Input is an optimized query plan

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Large Inputs-PCIe included 31 Average 2.22x speedup overall and 2.35x speedup in PCIe Large inputs (1GB-1.6GB) fitting the GPU memory

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Resource Usage & Occupancy 32 PTX Reg #Shared MEM (Byte) Occupancy (%) PROJECT SELECT JOIN / Multiply PTX Reg #Shared MEM (Byte) Occupancy (%) (a) (b) (c) (d) (e)27075 Kernel fusion may increase resource usage and thus decrease occupancy These two do not negate the other benefits Individual primitive After kernel fusion

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Real Queries (scale factor = 1) 33 TPC-H Q1 1.25x speedup TPC-H Q x speedup

SCHOOL OF ELECTRICAL AND COMPUTER ENGINEERING | GEORGIA INSTITUTE OF TECHNOLOGY Extensions Different Domains Require multi-stage algorithm Dependence classification still applies Different Representation PTX, OpenCL, LLVM Different Platform CPU, GPU/CPU hybrid 34