Download presentation

Presentation is loading. Please wait.

Published byMatthew South Modified over 3 years ago

1
**Vector: Data Layout Vector: x[n] P processors Assume n = r * p**

A[m:n] for(i=m;i<=n) A[i]… Let A[m : s : n] denotes for(i=m;i<=n;i=i+s) A[i] … Block distribution: id = 0, 1, …, p-1 x[r*id : r*(id+1)-1] id-th processor Cyclic distribution: x[id : p : n-1] id-th processor Block cyclic distribution: x = [x1, x2, …, xN]^T where xj is a subvector of length n/N Assume N = s*p Do a cyclic distribution of xj, j=1,2…,N n/N P*n/N n Block cyclic r n block p n cyclic

2
**Matrix: Data Layout Row: block, cyclic, block cyclic**

Column: block, cyclic, block cyclic Matrix: 9 combinations If only one block in one direction 1D decomposition Otherwise 2D decomposition e.g. p processors with a (Nx, Ny) virtual topology, p=Nx*Ny Matrix A[n][n], n = rx * Nx = ry * Ny A[rx*I : rx*(I+1)-1, J:Ny:n-1], I=0,…,rx-1, J=0,…,ry-1, is a 2D decomposition, block distribution in x direction, cyclic distribution in y direction 1D block cyclic 2D block cyclic

3
**Matrix-Vector Multiply: Row-wise**

= A X Y A11 A12 A13 A21 A22 A23 A31 A32 A33 X1 X2 X3 Y1 Y2 Y3 AX=Y A – NxN matrix, row-wise block distribution X,Y – vectors, dimension N cpu 0 cpu 1 cpu 2 Y1 = A11*X1 + A12*X2 + A13*X3 Y2 = A21*X1 + A22*X2 + A23*X3 Y3 = A31*X1 + A32*X2 + A33*X3 = A11 A12 A13 A21 A22 A23 A31 A32 A33 X2 X3 X1 Y1 Y2 Y3 cpu 0 cpu 1 cpu 2 Y1 = A11*X1 + A12*X2 + A13*X3 Y2 = A21*X1 + A22*X2 + A23*X3 Y3 = A31*X1 + A32*X2 + A33*X3 = A11 A12 A13 A21 A22 A23 A31 A32 A33 X3 X1 X2 Y1 Y2 Y3 cpu 0 cpu 1 cpu 2 Y1 = A11*X1 + A12*X2 + A13*X3 Y2 = A21*X1 + A22*X2 + A23*X3 Y3 = A31*X1 + A32*X2 + A33*X3

4
**Matrix-Vector Multiply: Column-wise**

= A X Y A11 A12 A13 A21 A22 A23 A31 A32 A33 X1 X2 X3 Y1 Y2 Y3 cpu 0 cpu 1 cpu 2 AX=Y A – NxN matrix, column-wise block distribution X,Y – vectors, dimension N Y1 = A11*X1 + A12*X2 + A13*X3 Y2 = A21*X1 + A22*X2 + A23*X3 Y3 = A31*X1 + A32*X2 + A33*X3 = A11 A12 A13 A21 A22 A23 A31 A32 A33 X1 X2 X3 Y2 Y3 Y1 cpu 0 cpu 1 cpu 2 Y2 = A21*X1 + A22*X2 + A23*X3 Y3 = A31*X1 + A32*X2 + A33*X3 Y1 = A11*X1 + A12*X2 + A13*X3 = A11 A12 A13 A21 A22 A23 A31 A32 A33 X1 X2 X3 Y3 Y1 Y2 cpu 0 cpu 1 cpu 2 Y3 = A31*X1 + A32*X2 + A33*X3 Y1 = A11*X1 + A12*X2 + A13*X3 Y2 = A21*X1 + A22*X2 + A23*X3

5
**Matrix-Vector Multiply: Row-wise**

All-gather

6
**Matrix-Vector Multiply: Column-wise**

= A X Y A11 A12 A13 A21 A22 A23 A31 A32 A33 X1 X2 X3 Y1 Y2 Y3 cpu 0 cpu 1 cpu 2 AX=Y A – NxN matrix, column-wise block distribution X,Y – vectors, dimension N First local computations Y1’ = A11*X1 Y2’ = A21*X1 Y3’ = A31*X1 Y1’ = A12*X2 Y2’ = A22*X2 Y3’ = A32*X2 Y1’ = A13*X3 Y2’ = A23*X3 Y3’ = A33*X3 Then reduce-scatter across processors

7
**Matrix-Vector Multiply: 2D Decomposition**

P=K^2 number of cpus As a 2D KxK mesh A – K x K block matrix, each block (N/K)x(N/K) X – K x 1 blocks, each block (N/K)x1 Each block of A is distributed to a cpu X is distributed to the K cpus in last column Result A*X be distributed on the K cpus of the last column P_{0} P_{1} … P_{K-1} P_{K} P_{K+1} P_{2K-1}

8
**Matrix-Vector Multiply: 2D Decomposition**

9
Homework Write an MPI program and implement the matrix vector multiplication algorithm with 2D decomposition Assume: Y=A*X, A – NxN matrix, X – vector of length N Number of processors P=K^2, arranged as a K x K mesh in a row-major fashion, i.e. cpus 0, …, K-1 in first row, K, …, 2K-1 in 2nd row, etc N can be divided by K. Initially, each cpu has the data for its own submatrix of A; Input vector X is distributed on processors of the rightmost column, i.e. cpus K-1, 2K-1, …, P-1 In the end, the result vector Y should be distributed on processors at the rightmost column. A[i][j] = 2*i+j, X[i] = i; Make sure your result is correct using a small value of N Turn in: Source code + binary Wall-time and speedup vs. cpu for 1, 4, 16 processors for N = 1024.

10
**Load Balancing: (Block) Cyclic**

x1 x2 x3 x4 x5 x6 x7 x8 x9 = y1 y4 y7 y2 y5 y8 y3 y6 y9 a b c x1 x2 x3 x4 x5 x6 x7 x8 x9 =

11
**Matrix-vector multiply, row-wise cyclic distribution of A and y**

Block distribution of x Initial data: id – cpu id p – number of cpus ids of left/right neighbors n – matrix dimension, n=r*p Aloc = A(id:p:n-1,:) yloc = y(id:p:n-1) xloc = x(id*r:(id+1)*r-1) Cyclic Distribution r = n/p for t=0:p-1 send(xloc,left) s = (id+t)%p // xloc = x(s*r:(s+1)*r-1) for i=0:r-1 for j=0:min(id+i*p-s*r,r) yloc(j) += Aloc(i,j+s*r)*xloc(j) end recv(xloc,right)

12
**Matrix Multiplication**

A – m x p B – p x n C – m x n C = AB + C Row: A(i,:) = [Ai1, Ai2, …, Ain] Column: A(:,j) = [A1j, A2j, …, Anj]^T Submatrix: A(i1:i2,j1:j2) = [ A(i,j) ], i1<=i<=i2, j1<=j<=j2 for i=1:m for j=1:n for k=1:p C(i,j) = C(i,j)+A(i,k)*B(k,j) end (ijk) variant of matrix multiplication for i=1:m for j=1:n C(i,j) = C(i,j)+A(i,:)B(:,j) end Dot product formulation A(i,:) dot B(:,j) A accessed by row B accessed by column Non-optimal memory access!

13
**Matrix Multiply ijk loop can be arranged in other orders**

(ikj) variant for i=1:m for k=1:p for j=1:n C(i,j) = C(i,j) + A(i,k)B(k,j) end axpy formulation B by row C by row for i=1:m for k=1:p C(i,:) = C(i,:) + A(i,k)B(k,:) end (jki) variant for j=1:n for k=1:p for i=1:m C(i,j) = C(i,j)+A(i,k)B(k,j) end for j=1:n for k=1:p C(:,j) = C(:,j)+A(:,k)B(k,j) end axpy formulation A by column C by column

14
**Other Variants Loop order Inner loop Middle loop**

Inner loop data access ijk Dot Axpy A by row, B by column jik ikj axpy B by row, C by row jki A by column, C by column kij Row outer product kji Column outer product

15
**Block Matrices also other variants Cache blocking**

Block matrix multiply A, B, C – NxN block matrices each block: s x s (mnp) variant for m=1:N for n = 1:N for p = 1:N i=(m-1)s+1 : ms j = (n-1)s+1 : ns k = (p-1)s+1 : ps C(i,j) = C(i,j) + A(i,k)B(k,j) end also other variants Cache blocking

16
Block Matrices

17
**Matrix Multiply: Column-wise**

B C A1 A2 A3 B11 B12 B13 B21 B22 B23 B31 B32 B33 C1 C2 C3 = C1 = A1*B11 + A2*B21 + A3*B cpu 0 C2 = A1*B12 + A2*B22 + A3*B cpu 1 C3 = A1*B13 + A2*B23 + A3*B cpu 2 A, B, C – NxN matrices P – number of processors A1, A2, A3 – Nx(N/P) matrices C1, C2, C3 - … Bij – (N/P)x(N/P) matrices Column-wise decomposition

18
**Matrix Multiply: Row-wise**

B1 B2 B3 C1 C2 C3 = C1 = A11*B1 + A12*B2 + A13*B3 cpu 0 C2 = A21*B1 + A22*B2 + A23*B3 cpu 1 C3 = A31*B1 + A32*B2 + A33*B3 cpu 2 A, B, C – NxN matrices P – number of processors B1, B2, B3 – (N/P)xN matrices C1, C2, C3 - … Aij – (N/P)x(N/P) matrices

19
**Matrix Multiply: 2D Decomposition Hypercube-Ring**

Cpus: P = K^2 Matrices A, B, C: dimension N x N, K x K blocks Each block: (N/K) x (N/K) Determine coordinate (irow,icol) of current cpu. Set B’=B_local For j=0:K-1 root_col = (irow+j)%K broadcast A’=A_local from root cpu (irow,root_col) to other cpus in the row C_local += A_local*B_local shift B’ upward one step end broadcast Broadcast A diagonals Shift B C fixed Step 1 Step 2 A01 A12 A23 A30 shift Step 2

20
Matrix Multiply Total ~K*log(K) communication steps, or sqrt(P)log(sqrt(P)) steps In contrast, 1D decomposition, P communication steps Can use max N^2 processors for problem size NxN matrices 1D decomposition, max N processors

21
**Matrix Multiply: Ring-Hypercube**

Determine coordinate (irow,icol) of current cpu. Set A’=A_local For j=0:K-1 root_row = (icol+j)%K broadcast B’=B_local from root cpu (root_row,icol) to other cpus in the column C_local += A_local*B_local Shift A’ leftward one step end Shift A columns Broadcast B diag C fixed Step 1 Step 2 A00 B00 A01 B11 A02 B22 A03 B33 A10 A11 A12 A13 A20 A21 A22 A23 A30 A31 A32 A33 A01 B10 A02 B21 A03 B32 A00 B03 A11 A12 A13 A10 A21 A22 A23 A20 A31 A32 A33 A30 Number of cpus: P=K^2 A, B, C: K x K block matrices each block: (N/K) x (N/K)

22
**Matrix Multiply: Ring-Hypercube**

compute initial Broadcast B Shift A broadcast compute Shift A Broadcast B A01 B10 A02 B21 A03 B32 A00 B03 A11 A12 A13 A10 A21 A22 A23 A20 A31 A32 A33 A30 A02 A03 A00 A01 A12 A13 A10 A11 A22 A23 A20 A21 A32 A33 A30 A31 A02 B20 A03 B31 A00 B02 A01 B13 A12 A13 A10 A11 A22 A23 A20 A21 A32 A33 A30 A31 Matrix Multiply: Ring-Hypercube

23
**Matrix Multiply: Systolic (Torus)**

B00 A01 B01 A02 B02 A10 B10 A11 B11 A12 B12 A20 B20 A21 B21 A22 B22 A B C fixed Number of cpus: P=K^2 A, B, C: K x K block matrices each block: (N/K) x (N/K) Shift rows of A leftward Shift columns of B upward initial Step 1 Step 2 Step 3 A00 B00 A01 B11 A02 B22 A11 B10 A12 B21 A10 B02 A22 B20 A20 B01 A21 B12 A01 B10 A02 B21 A00 B02 A12 B20 A10 B01 A11 B12 A20 B00 A21 B11 A22 B22 A02 B20 A00 B01 A01 B12 A10 B00 A11 B11 A12 B22 A21 B10 A22 B21 A20 B02

24
**Matrix Multiply: Systolic**

P = K^2 number of processors, as a K x K 2D torus A, B, C: KxK block matrices, each block (N/K)x(N/K) Each cpu computes 1 block: A_loc, B_loc, C_loc Coordinate in torus of current cpu: (irow, icol) Ids of left, right, top, bottom neighboring processors // first get appropriate initial distribution for j=0:irow-1 send(A_loc,left); recv(A_loc,right) end for j=0:icol-1 send(B_loc,top); recv(B_loc,bottom) // start computation for j=0:K-1 send(A_loc,left) send(B_loc,top) C_loc = C_loc + A_loc*B_loc recv(A_loc,right) recv(B_loc,bottom) Max N^2 processors ~ sqrt(P) communication steps

25
**Matrix Multiply on P=K^3 CPUs**

Assume: A, B, C: dimension N x N P = K^3 number of processors Organized into K x K x K 3D mesh A (NxN) can be considered as a q x q block matrix, each block (N/q)x(N/q) Let q = K^(1/3), i.e. consider A as a K^(1/3) x K^(1/3) block matrix, each block being (N/K^(1/3)) x (N/K^(1/3)) Similar for B and C

26
**Matrix Multiply on K^3 CPUs**

r,s = 1, 2, …, K^(1/3) Total K^(1/3)*K^(1/3)*K^(1/3) = K block matrix multiplications Idea: Perform these K matrix multiplications on the K different planes (or levels) of the 3D mesh of processors. Processor (i,j,k) (i,j=1,…,K) belongs to plane k. Will perform multiplication A_{rt}*B_{ts}, where k = (r-1)*K^(2/3)+(s-1)*K^(1/3)+t Within a plane, (N/K^(1/3)) x (N/K^(1/3)) matrix multiply on K x K processors. Use the systolic multiplication algorithm. Within a plane k: A_{rt}, B_{ts} and C_{rs} decomposed into K x K block matrices, each block (N/K^(4/3)) x (N/K^(4/3)).

27
**Matrix Multiply B_{ts} A_{rt} x On KxK processors A, B, C**

(i,j) On KxK processors N/K^(1/3) dimension K^(1/3) blocks K blocks A, B, C A_{rt} destined to levels k=(r-1)*K^(2/3)+(s-1)K^(1/3)+t, for all s=1,…,K^(1/3) B_{ts} destined to levels k=(r-1)*K^(2/3)+(s-1)*K^(1/3)+t, for all r=1,…,K^(1/3) Initially, processor (i,j,1) has (i,j) sub-blocks of all A_{rt} and B_{ts} blocks, for all r,s,t=1,…,K^(1/3), i,j=1,…,K Initial data distribution

28
**Matrix Multiply // set up input data On processor (i,j,1),**

read in the (i,j)-th block of matrices A_{r,t} and B_{t,s}, 1<= r,s,t <= K^(1/3); pass data onto processor (i,j,2); On processor (i,j,m), make own copy of A_{rt} if m=(r-1)*K^(2/3)+(s-1)*K^(1/3)+t for some s=1,...,K^(1/3); make own copy of B_{ts} if m=(r-1)*K^(2/3)+(s-1)*K^(1/3)+t for some r=1,...,K^(1/3); pass data onward to (i,j,m+1); // Computation On each processor (i,j,m), Compute A_{rt}*B_{ts} on the K x K processors using the systolic matrix multiplication algorithm; Some initial data setup may be needed before multiplication; // Summation Determine (r0,s0) of matrix the current processor (i,j,k) works on: r0 = k/K^(2/3)+1; s0 = (k-(r0-1)*K^(2/3))/K^(1/3); Do reduction (sum) over processors (i,j,m), m=(r0-1)*K^(2/3)+(s0-1)*K^(1/3)+t, of all 1<=t<=K^(1/3);

29
**Matrix Multiply Communication steps: ~K, or P^(1/3)**

Maximum CPUs: N/K^(4/3) = 1 K=N^(3/4), or P=N^(9/4)

30
Matrix Multiply If number of processors: P = KQ^2, arranged into KxQxQ mesh K planes Each plane QxQ processors Handle similarly Decompose A, B, C into K^(1/3)xK^(1/3) blocks Different block matrix multiplications in different planes, K multiplications total Each block multiplication handled in a plane on QxQ processors; use any favorable algorithm, e.g. systolic

31
**Processor Array in Higher Dimension**

Processors P=K^4, arranged into KxKxKxK mesh Similar strategy: Divide A,B,C into K^(1/3)xK^(1/3) block matrices Different multiplications (total K) computed on different levels of 1st dimension Each block matrix multiplication done on the KxKxK mesh at one level; repeat the above strategy. For even higher dimensions, P=K^n, n>4, handle similarly.

32
**Matrix Multiply: DNS Algorithm**

Assume: A, B, C: dimension N x N P = K^3 number of processors Organized into K x K x K 3D mesh A, B, C are K x K block matrices, each block (N/K) x (N/K) Total K*K*K block matrix multiplications Idea: each block matrix multiplication is assigned to one processor Processor (i,j,k) computes C_{ij}=A_{ik}*B_{kj} Need a reduction (sum) over processors (i,j,k), k=0,…,K-1

33
**Matrix Multiply: DNS Algorithm**

Initial data distribution: A_{ij} and B_{ij} at processor (i,j,0) Need to trasnsfer A_{ik} (i,k=0,…,K-1) to processor (i,j,k) for all j=0,1,…,K-1 Two steps: - Send A_{ik} from processor (i,k,0) to (i,k,k); - Broadcast A_{ik} from processor (i,k,k) to processors (i,j,k);

34
**Matrix Multiply Send A_{ik} from (i,k,0) to (i,k,k)**

To broadcast A_{ik} from (i,k,k) to (i,j,k)

35
**Matrix Multiply Final data distribution for A**

A can also be considered to come in through the (i,k) plane; with a broadcast along the j-direction.

36
**B Distribution B distribution: Initially B_{kj} in processor (k,j,0);**

Need to transfer to processors (i,j,k) for all i=0,1,…,K-1 Two steps: First send B_{kj} from (k,j,0) to (k,j,k) Broadcast B_{kj} from (k,j,k) to (i,j,k) for all i=0,…,K-1, i.e. along i-direction

37
**B Distribution Send B_{kj} from (k,j,0) to (k,j,k)**

0,0 0,1 0,2 0,3 1,0 1,1 1,2 1,3 2,0 2,1 2,2 2,3 3,0 3,1 3,2 3,3 i j k B Distribution Send B_{kj} from (k,j,0) to (k,j,k) To broadcast from (k,j,k) to along i direction

38
**Matrix Multiply Final B distribution:**

B can also be considered to come through (j,k) plane; then broadcast along i-direction

39
**Matrix Multiply A_{ik} and B_{kj} on cpu (i,j,k)**

Compute C_{ij} locally Reduce (sum) C_{ij} along k-direction Final result: C_{ij} on cpu (i,j,0) Matrix Multiply

40
Matrix Multiply A matrix comes through (i,k) plane, broadcast along j-direction B matrix comes through (j,k) plane, broadcast along i-direction C matrix result goes to (i,j) plane Broadcast: 2log(K) steps Reduction: log(K) steps Total: 3log(K) = log(P) steps Can use a maximum of P=N^3 processors In contrast: Systolic: P^(1/2) communication steps Can use a maximum of P=N^2 processors Slide #24: P^(1/3) communication steps Can use a maximum of P=N^(9/4) processors

Similar presentations

OK

Velocity and Acceleration. Definitions of Velocity and Acceleration.

Velocity and Acceleration. Definitions of Velocity and Acceleration.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google