1. Communication operations Efficient Parallel Algorithms COMP308

2. Communication time Communication requires 3 costs: 1. Static start up time (t s ): –It is the time required to handle a message at the sending processor 2. Per-hop time (t h ) with l the #Links that the message passes –It is take a finite amount of time to reach the next processor in its path after a message leaves a processor. 3. Per-word transfer time (t w ): with m the #bytes –If the channel bandwidth is r words per second, then each word takes time t w =1/r to traverse the link.

3. There are 2 main communication schemes:

4. “store and forward” vs “cut-through” In “store and forward” routing, when a message is traversing a path with multiple links, each intermediate node on the path forwards the message to the next node after it has received. In “cut-through” routing an intermediate nodes does not wait for the entire message to arrive before forwarding it. –A tracer is first sent from the source to the designation node to establish a connection. –Once a connection has been established, the flits are sent one after the other. All flits follows the same path in a dovetailed fashion. –As soon as a flit is received at an intermediate node, the flit is passed on to the next node.

5. One to All Broadcast Initially, only the source processor has the data of size m that need to be broadcast. At the end of the termination of the procedure, there are P copies of the initial data, one residing at each processor.

6. Broadcast on ring (Store and Forward) If the sender sends the messages consecutively to the p-1 other processors, it takes p-1 steps. By optimisation, we can reduce this to p/2 steps. Eg.: a 8-processor ring requires 4 steps

7. NS diagram for “broadcast on ring”

8. Ring network, Cut-Through routing With cut-through routing, messages can be sent faster to nodes that are multiple hops away in the network. By using this, we send the message first to the outermost node. In general, in a p-processor ring the source processor first sends the data to the processor at distance p/2, then both processors sends the message to the processors at distance of p/4 in the same direction, then to p/8, etc.

9. Broadcast on mesh (Store and Forward) Most of the optimised communication algorithms on a mesh are simple extensions of their ring counterparts, by consecutively applying the ring algorithm on each dimension of the mesh.

10. Hypercube The regular binary structure of the hypercube plays an important role in optimising communication. Here, a broadcast is performed by sending the message along each dimension at each step. This results in log p or d steps. It can be proved easily that log p is the minimal number of steps for every network.

11. Hypercube Important properties of the networks: –Small degree, –Small diameter, –Regular recursive structure, –Easy way to embed trees, etc Hypercube – two nodes connected if they are differ precisely on one bit

13. 01 00101 10101 000001 010011 100101 110111 00000001 00100011 01000101 01100111 10001001 10101011 11001101 11101111

14. 1000001 1010011 11001101 11101111 00000001 00100011 01000101 01100111

16. Broadcast on hypercube (S&F)

17. Broadcast on ring (Cut-Through )

18. Broadcast on mesh (C-T)

19. Broadcast on binary tree (C-T)

20. Gossiping All-to-All Communication

21. Gossiping on Ring (Store and Forward)

22. Gossiping on Mesh (Store and Forward)

23. Gossiping on Hypercube (S&F)

24. Gossiping on Ring (and Mesh) Cut-Through Routing Each process sends m(p-1) words of data because it has an m- word packet for every other processor The average distance that an m word packet travels is Since there are p processors, each performing the same type of communication, the total traffic on the network is The total number of communication channels in the network to share this load is p. Hence this procedure cannot be improved by using CT routing

25. Gossiping on Hypercube (CT routing)