Presentation is loading. Please wait.

Presentation is loading. Please wait.

SuperMatrix on Heterogeneous Platforms Jianyu Huang SHPC, UT Austin 1.

Published byMiles Owens Modified over 8 years ago

Similar presentations

Presentation on theme: "SuperMatrix on Heterogeneous Platforms Jianyu Huang SHPC, UT Austin 1."— Presentation transcript:

1 SuperMatrix on Heterogeneous Platforms Jianyu Huang SHPC, UT Austin 1

2 How Heterogeneous? 2

3 How Many Languages? 3

4 Question! 4

5 FLAME Answer: SuperMatrix Programmability  Use tools provide by FLAME Ernie Chan. “Application of Dependence Analysis and Runtime Data Flow Graph Scheduling to Matrix Computations.” Ph.D. dissertation, Department of Computer Science, The University of Texas at Austin Chan, Ernie, et al. "SuperMatrix out-of-order scheduling of matrix operations for SMP and multi-core architectures." Proceedings of the nineteenth annual ACM symposium on Parallel algorithms and architectures. ACM, 2007. 5 Parallelism  Directed acyclic graph (DAG) scheduling SuperMatrix libflame clBLAS OpenCL cuBLAS CUDA GPUCPU/MIC ACML C/asse mbly/F ortran MKL C/For tran/ Asse mbly BLIS C/Asse mbly Accelerator/Other platforms BLIS C/Asse mbly OpenMP /pthread

6 FLAME Answer: SuperMatrix Chan, E., Quintana-Ortí, E. S., Quintana-Ortí, G., and van de Geijn, R.. SuperMatrix out-of-order scheduling of matrix operations for SMP and multi-core architectures. In SPAA'07: Proceedings of the Nineteenth Annual ACM Symposium on Parallelism in Algorithms and Architectures, pages 116-125, San Diego, CA, USA, June 9-11, 2007. Chan, E., G. Van Zee, F., Quintana-Ortí, E. S., Quintana-Ortí, G., and van de Geijn, R.. Satisfying your dependencies with SuperMatrix. InCluster'07: Proceedings of the 2007 IEEE International Conference on Cluster Computing, pages 91-99, Austin, TX, USA, September 17-20, 2007. Chan, E., G. Van Zee, F., Bientinesi, P., Quintana-Ortí, E. S., Quintana-Ortí, G., and van de Geijn, R.. SuperMatrix: A multithreaded runtime scheduling system for algorithms-by-blocks. In PPoPP'08: Proceedings of the Thirteenth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 123-132, Salt Lake City, UT, USA, February 20-23, 2008. Quintana-Orti, G., Igual, F. D., Quintana-Orti, E. S., van de Geijn, R.. Solving dense linear systems on platforms with multiple hardware accelerators. In PPoPP '09 Proceedings of the 14th ACM SIGPLAN symposium on Principles and Practice of Parallel Programming, 2009 Quintana-Ortí, G., Quintana-Ortí, E.S., van de Geijn, R., G. Van Zee, F., and Chan, E.. Programming matrix algorithms-by-blocks for thread-level parallelism. ACM Transactions on Mathematical Software, 36(3):14:1- 14:26, July 2009. Chan, E.. “Application of Dependence Analysis and Runtime Data Flow Graph Scheduling to Matrix Computations.” Ph.D. dissertation, Department of Computer Science, The University of Texas at Austin Quintana-Ortí, G., Igual, F. D., Marqués, M., Quintana-Ortí, E. S., and van de Geijn, R.. "A Runtime System for Programming Out-of-Core Matrix Algorithms-by-Tiles on Multithreaded Architectures." ACM Transactions on Mathematical Software (TOMS) 38, no. 4 (2012): 25. 6

7 Parallel? S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X Write After Read: (S0, S1) Are you sure S1 and S2 cannot be parallelized? Read After Write: (S1, S4) Read After Write: (S1, S2) Read After Write: (S2, S3) Can the code be parallelized? Read After Write: (S0, S1) 7

8 Parallel? S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? DAB A L BB L CBTBT BC X L B 8

9 Traditional Library Approach S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? S0: ParGemm (A,B,D) S1: L = ParPortf(A) S2: ParTrsm(L,B) S3: ParSyrk(B,C) S4: ParTrsm(L,X) 9

10 Traditional Library Approach Implemented with libflame and BLIS S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X Supported by parallel BLAS, LAPACK ( multi-thread BLIS ) /*-----------------------------------------------*/ FLA_Gemm( FLA_NO_TRANSPOSE,FLA_NO_TRANSPOSE, FLA_ONE, A, B, FLA_ZERO, D ); FLA_Chol( FLA_LOWER_TRIANGULAR, A ); FLA_Trsm( FLA_RIGHT, FLA_LOWER_TRIANGULAR, FLA_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, A, B ); FLA_Syrk( FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_MINUS_ONE, B, FLA_ONE, C ); FLA_Trsm( FLA_LEFT, FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, L, X ); /*-----------------------------------------------*/ 10

11 Problem for Fine-grained Parallelism libflame BLIS pthreads OpenMP Fine-grained parallelism libflame BLIS Coarse-grained parallelism BLIS pthreads OpenMP 11 Introduce parallelism across instructions Fit for the platform with multiple computation units. Synchronization point overhead Not fit for multiple devices scenarios.

12 Coarse-grained Parallelism libflame BLIS pthreads OpenMP Fine-grained parallelism libflame BLIS Coarse-grained parallelism BLIS SuperMatrix 12 Introduce parallelism across instructions Fit for the platform with multiple computation units.

13 SuperMatrix Approach S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? DAB A L BB L CBTBT BC X L B 13

14 SuperMatrix Approach S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? 14

15 SuperMatrix Approach S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? Partitioning/Algorithm-by-blocks! 15

16 SuperMatrix Approach S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X How to parallelize? 16

17 SuperMatrix Approach Construct the DAG across the instructions automatically No need to annotate the task dependencies manually! 17

18 Traditional Library Approach Implemented with libflame and BLIS Supported by parallel BLAS, LAPACK ( multi-thread BLIS ) /*-----------------------------------------------*/ FLA_Gemm( FLA_NO_TRANSPOSE,FLA_NO_TRANSPOSE, FLA_ONE, A, B, FLA_ZERO, D ); FLA_Chol( FLA_LOWER_TRIANGULAR, A ); FLA_Trsm( FLA_RIGHT, FLA_LOWER_TRIANGULAR, FLA_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, A, B ); FLA_Syrk( FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_MINUS_ONE, B, FLA_ONE, C ); FLA_Trsm( FLA_LEFT, FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, L, X ); /*-----------------------------------------------*/ 18 S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X

19 SuperMatrix Approach Implemented with libflame and BLIS S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X /*-----------------------------------------------*/ FLASH_Gemm( FLA_NO_TRANSPOSE,FLA_NO_TRANSPOSE, FLA_ONE, A, B, FLA_ZERO, D ); FLASH_Chol( FLA_LOWER_TRIANGULAR, A ); FLASH_Trsm( FLA_RIGHT, FLA_LOWER_TRIANGULAR, FLA_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, A, B ); FLASH_Syrk( FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_MINUS_ONE, B, FLA_ONE, C ); FLASH_Trsm( FLA_LEFT, FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, L, X ); /*-----------------------------------------------*/ 19

20 Free Lunch for Both Programmability and Performance! From libflame manual, 2011 20

21 Original SuperMatrix primarily targets at multi-core shared memory system… 21

22 HPC Heterogeneous Platforms PCIE 22 Matrix

23 Challenges in Heterogeneous Platforms! S0: D ← A*A T S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X What if there is one accelerator in your system? S0: ParGemm (A,A T,D) S1: L = ParPortf(A) S2: ParTrsm(L,B) S3: ParSyrk(B,C) S4: ParTrsm(L,X) 23

24 Challenges in Heterogeneous Platforms! S0: ParGemm (A,A T,D) S1: L = ParPortf(A) S2: ParTrsm(L,B) S3: ParSyrk(B,C) S4: ParTrsm(L,X) What if there are 4 GPUs and 8 CPU cores in your system? /*-----------------------------*/ Memcpy(A, hA); Memcpy(D, hD); Memcpy(B, hB); Memcpy(C, hC); Memcpy(X, hX); /*-----------------------------*/ /*-----------------------------*/ Memcpy(hX, X); /*-----------------------------*/ What if there is one accelerator in your system? 24

25 Adapting Original SuperMatrix to Heterogeneous Platforms Software Cache Heterogeneous Scheduler Asynchronous Memory Copy Worker Task Performance Model 25

26 Naïve Approach PCIE CAB Transfer data from host to device before execution Transfer data from device to host upon execution Execute the task on the device No Data Reuse on the devices ! 26

27 Software Cache PCIE Quintana-Ortí, G., et al. "Solving dense linear systems on platforms with multiple hardware accelerators." In PPoPP '09 Proceedings of the 14th ACM SIGPLAN symposium on Principles and Practice of Parallel Programming, 2009 CAB No need to transfer data from host to device before execution if the data is already on the device No need to transfer data from device to host upon execution if the data is not required by the host immediately 27

28 HEFT(Heterogeneous Earliest Finish Time) Topcuoglu, H., Hariri, S., and Wu, M.. "Performance-effective and low-complexity task scheduling for heterogeneous computing." IEEE Transactions on Parallel and Distributed Systems, 13.3 (2002): 260-274. Timeline 8 Task 2 Task 1 Task 3Task 4 Task 5 Task 6 Where should we place Task 6? Task 6 28

29 3x3 Blocked Cholesky Decomposition CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) 29

30 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00100 A10100 A11100 A20100 A21100 A22100 CPUGPU0GPU1 CHOL0 TRSM1 TRSM2 SYRK3 GEMM4 SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail000 EST EFT Priority HEFT assignment table 30

31 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00100 A10100 A11100 A20100 A21100 A22100 CPUGPU0GPU1 CHOL0 X TRSM1 TRSM2 SYRK3 GEMM4 SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail000 EST011 EFT1.522 Priority123 HEFT assignment table 31

32 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00110 A10010 A11100 A20100 A21100 A22100 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 SYRK3 GEMM4 SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.500 EST1.53.5 EFT5.555 Priority312 HEFT assignment table 32

33 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11100 A20001 A21100 A22100 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 GEMM4 SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.550 EST1.553.5 EFT5.56.55 Priority231 HEFT assignment table 33

34 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11010 A20001 A21100 A22100 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.555 EST657 EFT106.58.5 Priority312 HEFT assignment table 34

35 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11010 A20111 A21010 A22100 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.56.55 EST677 EFT1410 Priority312 HEFT assignment table 35

36 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11010 A20111 A21010 A22001 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 X CHOL6 TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.5105 EST6105 EFT1011.56.5 Priority231 HEFT assignment table 36

37 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11100 A20111 A21010 A22001 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 X CHOL6 X TRSM7 SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail1.5106.5 EST7.5108.5 EFT9119.5 Priority132 HEFT assignment table 37

38 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11110 A20111 A21010 A22001 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 X CHOL6 X TRSM7 X SYRK8 CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail9106.5 EST111012 EFT1511.513.5 Priority312 HEFT assignment table 38

39 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11110 A20111 A21010 A22010 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 X CHOL6 X TRSM7 X SYRK8 X CHOL9 Data Distribution Scheduler CPUGPU0GPU1 Avail911.56.5 EST12.511.513.5 EFT16.51315 Priority312 HEFT assignment table 39

40 CHOL 0 A 0,0 ← Chol( A 0,0 ) CHOL 0 A 0,0 ← Chol( A 0,0 ) TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 2 A 2,0 ← A 2,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T TRSM 1 A 1,0 ←A 1,0 A 0,0 -T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T SYRK 5 A 2,2 ← A 2,2 – A 2,0 A 2,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T GEMM 4 A 2,1 ← A 2,1 – A 2,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T SYRK 3 A 1,1 ← A 1,1 – A 1,0 A 1,0 T CHOL 6 A 1,1 ← Chol( A 1,1 ) CHOL 6 A 1,1 ← Chol( A 1,1 ) TRSM 7 A 2,1 ← A 2,1 A 1,1 -T TRSM 7 A 2,1 ← A 2,1 A 1,1 -T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T SYRK 8 A 2,2 ← A 2,2 – A 2,1 A 2,1 T CHOL 9 A 2,2 ← Chol( A 2,2 ) CHOL 9 A 2,2 ← Chol( A 2,2 ) CPUGPU0GPU1 A00111 A10010 A11110 A20111 A21010 A22010 CPUGPU0GPU1 CHOL0 X TRSM1 X TRSM2 X SYRK3 X GEMM4 X SYRK5 X CHOL6 X TRSM7 X SYRK8 X CHOL9 X Data Distribution Scheduler CPUGPU0GPU1 Avail9136.5 EST141315 EFT15.51416 Priority213 HEFT assignment table 40

41 SuperMatrix Approach on Heterogeneous Platforms S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X /*-----------------------------------------------*/ FLASH_Gemm( FLA_NO_TRANSPOSE,FLA_NO_TRANSPOSE, FLA_ONE, A, B, FLA_ZERO, D ); FLASH_Chol( FLA_LOWER_TRIANGULAR, A ); FLASH_Trsm( FLA_RIGHT, FLA_LOWER_TRIANGULAR, FLA_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, A, B ); FLASH_Syrk( FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_MINUS_ONE, B, FLA_ONE, C ); FLASH_Trsm( FLA_LEFT, FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, L, X ); /*-----------------------------------------------*/ 41

42 Performance 42 6-core single-socket Xeon E5649 CPU + 1 GTX 480 GPU card BLOCK SIZE: 1024 6-core single-socket Xeon E5649 CPU + 2 Tesla C2070 GPU card BLOCK SIZE: 2048

43 SuperMatrix libflame clBLAS OpenCL cuBLAS CUDA GPUCPU/MIC ACML C/asse mbly/F ortran MKL C/For tran/ Asse mbly BLIS C/Asse mbly Accelerator/Other platforms BLIS C/Asse mbly Conclusion OpenMP /pthread 43

44 SuperMatrix Approach on Heterogeneous Platforms S0: D ← A*B S1: A → L * L T S2: B ← B * L -T S3: C ← C – B * B T S4: X ← L -1 * X /*-----------------------------------------------*/ FLASH_Gemm( FLA_NO_TRANSPOSE,FLA_NO_TRANSPOSE, FLA_ONE, A, B, FLA_ZERO, D ); FLASH_Chol( FLA_LOWER_TRIANGULAR, A ); FLASH_Trsm( FLA_RIGHT, FLA_LOWER_TRIANGULAR, FLA_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, A, B ); FLASH_Syrk( FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_MINUS_ONE, B, FLA_ONE, C ); FLASH_Trsm( FLA_LEFT, FLA_LOWER_TRIANGULAR, FLA_NO_TRANSPOSE, FLA_NONUNIT_DIAG, FLA_ONE, L, X ); /*-----------------------------------------------*/ 44

45 Related Work Target PlatformLapack ProjectFLAME Project SequentialLAPACKlibflame Sequential+multithreaded BLASLAPACKlibflame Multicore/multithreadedPLASMAlibflame+SuperMatrix Multicore+out-of-order schedulingPLASMA+Quarklibflame+SuperMatrix CPU + single GPUMAGMAlibflame+SuperMatrix Multicore + multi-GPUDAGuE/StarPU/ XKaapi libflame+SuperMatrix 45

46 Questions? 46

Download ppt "SuperMatrix on Heterogeneous Platforms Jianyu Huang SHPC, UT Austin 1."

Similar presentations

Prasanna Pandit R. Govindarajan

Prasanna Pandit R. Govindarajan
Dense Linear Algebra (Data Distributions) Sathish Vadhiyar.

Dense Linear Algebra (Data Distributions) Sathish Vadhiyar.
Motivation Desktop accelerators (like GPUs) form a powerful heterogeneous platform in conjunction with multi-core CPUs. To improve application performance.

Motivation Desktop accelerators (like GPUs) form a powerful heterogeneous platform in conjunction with multi-core CPUs. To improve application performance.
ACCELERATING MATRIX LANGUAGES WITH THE CELL BROADBAND ENGINE Raymes Khoury The University of Sydney.

ACCELERATING MATRIX LANGUAGES WITH THE CELL BROADBAND ENGINE Raymes Khoury The University of Sydney.
Scalable Data Clustering with GPUs Andrew D. Pangborn Thesis Defense Rochester Institute of Technology Computer Engineering Department Friday, May 14 th.

Scalable Data Clustering with GPUs Andrew D. Pangborn Thesis Defense Rochester Institute of Technology Computer Engineering Department Friday, May 14 th.
April 19, 2010HIPS 20101 Transforming Linear Algebra Libraries: From Abstraction to Parallelism Ernie Chan.

April 19, 2010HIPS Transforming Linear Algebra Libraries: From Abstraction to Parallelism Ernie Chan.
L15: Review for Midterm. Administrative Project proposals due today at 5PM (hard deadline) – handin cs6963 prop March 31, MIDTERM in class L15: Review.

L15: Review for Midterm. Administrative Project proposals due today at 5PM (hard deadline) – handin cs6963 prop March 31, MIDTERM in class L15: Review.
Lock vs. Lock-Free memory Fahad Alduraibi, Aws Ahmad, and Eman Elrifaei.

Lock vs. Lock-Free memory Fahad Alduraibi, Aws Ahmad, and Eman Elrifaei.
CUDA Programming Lei Zhou, Yafeng Yin, Yanzhi Ren, Hong Man, Yingying Chen.

CUDA Programming Lei Zhou, Yafeng Yin, Yanzhi Ren, Hong Man, Yingying Chen.
Linear Algebra on GPUs Vasily Volkov. GPU Architecture Features SIMD architecture – Don’t be confused by scalar ISA which is only a program model We use.

Linear Algebra on GPUs Vasily Volkov. GPU Architecture Features SIMD architecture – Don’t be confused by scalar ISA which is only a program model We use.
Big Kernel: High Performance CPU-GPU Communication Pipelining for Big Data style Applications Sajitha Naduvil-Vadukootu CSC 8530 (Parallel Algorithms)

Big Kernel: High Performance CPU-GPU Communication Pipelining for Big Data style Applications Sajitha Naduvil-Vadukootu CSC 8530 (Parallel Algorithms)
Parallelization and CUDA libraries Lei Zhou, Yafeng Yin, Hong Man.

Parallelization and CUDA libraries Lei Zhou, Yafeng Yin, Hong Man.
Contemporary Languages in Parallel Computing Raymond Hummel.

Contemporary Languages in Parallel Computing Raymond Hummel.
Panda: MapReduce Framework on GPU’s and CPU’s

Panda: MapReduce Framework on GPU’s and CPU’s
GPGPU overview. Graphics Processing Unit (GPU) GPU is the chip in computer video cards, PS3, Xbox, etc – Designed to realize the 3D graphics pipeline.

GPGPU overview. Graphics Processing Unit (GPU) GPU is the chip in computer video cards, PS3, Xbox, etc – Designed to realize the 3D graphics pipeline.
To GPU Synchronize or Not GPU Synchronize? Wu-chun Feng and Shucai Xiao Department of Computer Science, Department of Electrical and Computer Engineering,

To GPU Synchronize or Not GPU Synchronize? Wu-chun Feng and Shucai Xiao Department of Computer Science, Department of Electrical and Computer Engineering,
Motivation “Every three minutes a woman is diagnosed with Breast cancer” (American Cancer Society, “Detailed Guide: Breast Cancer,” 2006) Explore the use.

Motivation “Every three minutes a woman is diagnosed with Breast cancer” (American Cancer Society, “Detailed Guide: Breast Cancer,” 2006) Explore the use.
Data Partitioning on Heterogeneous Multicore and Multi-GPU Systems Using Functional Performance Models of Data-Parallel Applications Published in: Cluster.

Data Partitioning on Heterogeneous Multicore and Multi-GPU Systems Using Functional Performance Models of Data-Parallel Applications Published in: Cluster.
1 Programming Multicore Processors Aamir Shafi High Performance Computing Lab

1 Programming Multicore Processors Aamir Shafi High Performance Computing Lab
An approach for solving the Helmholtz Equation on heterogeneous platforms An approach for solving the Helmholtz Equation on heterogeneous platforms G.

An approach for solving the Helmholtz Equation on heterogeneous platforms An approach for solving the Helmholtz Equation on heterogeneous platforms G.

Similar presentations

Ads by Google