Towards Communication Avoiding Fast Algorithm for Sparse Matrix Multiplication Part I: Minimizing arithmetic operations Oded Schwartz CS294, Lecture #21 Fall, 2011 Communication-Avoiding Algorithms Based on: R. Yuster and U. Zwick Fast Sparse Matrix Multiplication G. Ballard, J. Demmel, O. Holtz, and O. Schwartz Communication Avoiding Fast Sparse Matrix Multiplication Many slides from:

2 How to multiply sparse matrices faster? Outline Strassen-like algorithms The naïve algorithm The Yuster-Zwick algorithm Again, minding communication costs: Fast rectangular matrix multiplication The naïve algorithm Retuning Yuster-Zwick algorithm

3 Matrix Multiplication n = C(i,j) =  k A(i,k) B(k,j) ABC ii jj

4 Strassen-like algorithms Compute n 0 x n 0 matrix multiplication using only n 0  0 multiplications (instead of n 0 3 ). Apply recursively (block-wise)  0  2.81[Strassen 69] works fast in practice. 2.79[Pan 78] 2.78[Bini 79] 2.55[Schönhage 81] 2.50 [Pan Romani,Coppersmith Winograd 84] 2.48 [Strassen 87] 2.38[Coppersmith Winograd 90] 2.38 [Cohn Kleinberg Szegedy Umans 05] Group-theoretic approach T(n) = n 0  0  T(n/n 0 ) + O(n 2 ) T(n) =  (n  0 ) n/n 0 = Are these faster on sparse matrices?

5 Sparse Matrix Multiplication  = n - number of rows and columns m - number of non-zero elements The distribution of the non-zero elements in the matrices is arbitrary!

6 Naïve Sparse Matrix Multiplication  = Each element of B is multiplied by at most n elements from A. Worst case complexity: mn j k k

7 Matrix multiplication AuthorsComplexity Coppersmith, Winograd (1990) n mn Is this as good as it gets? Yuster, Zwick (2005) m 0.7 n 1.2 +n 2+o(1)

8 Comparison r (m=n r )  n 2.38 mn m 0.7 n 1.2 +n 2 Complexity = n 

9 A closer look at the naïve algorithm  =  =

10 Complexity of the naïve algorithm Complexity = where Can it really be that bad? a i (and b i ) are the number of non-zeros in the i th column of A (row of B)

11 Regular case: Best case for naïve algorithm

12 Worst case for naïve algorithm  = 0 0 vs. m 2 /n (best case)

13 Rectangular Matrix multiplication How fast can we multiply (dense) rectangular matrices?  = n p p n n n

14 Fast Rectangular matrix multiplication Compute matrix multiplication on matrices of dimension a  b and b  c using only  =  (a, b, c) < abc multiplications Apply recursively (block-wise)  =  = a= 3 a 2 = 9 b = 2 b 2 = 4 c = 4 c 2 = 16

15 Fast Rectangular matrix multiplication [Brockett, Dobkin 76] T(n, n, log n) = n 2 + o(n 2 ) [Coppersmith 82] T(n, n, n  ) = O(n 2+  ) for  = [Coppersmith 97] T(n, n, n  ) = O(n 2+  ) for  = [Coppersmith Winograd 90] T(n, n, n 2 ) = O(n ) [Huang, Pan 97] T(n, n, n 2 ) = O(n ) [Coppersmith 97] T(n, n, p) = O(n 2+  + n 1.85 p 0.54 ) Is it better than black-box use of fast square matrix multiplication? n nn2n2 [Coppersmith Winograd 90] Dense square matrix multiplication in  (n 2.38 )

16 Fast Rectangular matrix multiplication [Brockett, Dobkin 76] T(n, n, log n) = n 2 + o(n 2 ) [Coppersmith 82] T(n, n, n  ) = O(n 2+  ) for  = [Coppersmith 97] T(n, n, n  ) = O(n 2+  ) for  = [Coppersmith Winograd 90] T(n, n, n 2 ) = O(n ) [Huang, Pan 97] T(n, n, n 2 ) = O(n ) [Coppersmith 97] T(n, n, p) = O(n 2+  + n 1.85 p 0.54 ) But Yuster-Zwick need T(n, p, n)… [Pan 72] T(a, b, c) = T(a, c, b) = T(b, a, c) = T(b, c, a) = T(c, a, b) = T(c, b, a) Which of these algorithm can we implement?

17 The combined algorithm Assume: a 1 b 1 ≥ a 2 b 2 ≥ … ≥ a n b n Choose: 0 ≤ p ≤ n Compute: AB = A 1 B 1 + A 2 B 2 Complexity: Fast rectangular matrix multiplication Naïve sparse matrix multiplication A1A1 A2A2 B1B1 B2B2

18 Analysis of combined algorithm Theorem: There exists a 1≤p≤n for which Lemma:

19 How to multiply sparse matrices faster? Outline Strassen-like algorithms The naïve algorithm The Yuster-Zwick algorithm Again, minding communication costs: The naïve algorithm Fast rectangular matrix multiplication Retuning Yuster-Zwick algorithm

Towards Communication Avoiding Fast Algorithm for Sparse Matrix Multiplication Oded Schwartz CS294, Lecture #21 Fall, 2011 Communication-Avoiding Algorithms Based on: R. Yuster and U. Zwick Fast Sparse Matrix Multiplication G. Ballard, J. Demmel, O. Holtz, and O. Schwartz Communication Avoiding Fast Sparse Matrix Multiplication Many slides from: Thank you!