Anshul Kumar, CSE IITD ECE729 : Advanced Computer Architecture Lecture 27, 28: Interconnection Mechanisms In Multiprocessors 29 th, 31 st March, 2010

Anshul Kumar, CSE IITD slide 2 Connecting Processors and Memories Shared Buses Interconnection Networks –Static Networks –Dynamic Networks PPPP MMM Interconnection Network M MMM PPPP MMM M MMM Global Interconnection Network MMM

Anshul Kumar, CSE IITD slide 3 Shared Bus each processor sees this picture: processing bus access prob of a processor using the bus =  prob of a processor not using the bus = 1 –  prob of none of the n processors using the bus = (1 –  ) n prob of at least one processor using the bus = 1 – (1 –  ) n achieved BW on a relative scale = 1 – (1 –  ) n required BW = n  available BW = 1

Anshul Kumar, CSE IITD slide 4 Effect of re-submitted requests A W  1-  Probability of a request being accepted P A Probability of a request not being accepted 1 - P A prob = q A prob = q W Active state Probability of making a bus request Probability of not making a bus request Wait state Probability of making a bus request 1 Probability of not making a bus request 0

Anshul Kumar, CSE IITD slide 5 Effect of re-submitted requests AW  (1-P A ) 1-  +  P A 1-P A PAPA prob = q A prob = q W

 without resubmission with resubmission

without resubmission with resubmission 

Anshul Kumar, CSE IITD slide 8 Waiting time

Anshul Kumar, CSE IITD slide 9 Buses vs. Networks Buses Shared media Lower Cost Lower throughput Scalability poor Networks Switched paths Higher cost Higher throughput Scalability better

Anshul Kumar, CSE IITD slide 10 Interconnection Networks Topology : who is connected to whom Direct / Indirect : where is switching done Static / Dynamic : when is switching done Circuit switching / packet switching : how are connections established Store & forward / worm hole routing : how is the path determined Centralized / distributed : how is switching controlled Synchronous/asynchronous : mode of operation

Anshul Kumar, CSE IITD slide 11 PMPM Direct and Indirect Networks PMSPMS PMSPMS SMPSMP SMPSMP PMPM PMPM PMPM SWITCH DIRECT INDIRECT node link node link

Anshul Kumar, CSE IITD slide 12 Static and Dynamic Networks Static Networks fixed point to point connections usually direct each node pair may not have a direct connection routing through nodes Dynamic Networks connections established as per need usually indirect path can be established between any pair of nodes routing through switches

Anshul Kumar, CSE IITD slide 13 Static Network Topologies Linear Star 2D-Mesh Tree Non-uniform connectivity

Anshul Kumar, CSE IITD slide 14 Static Networks Topologies- contd. Ring Fully Connected Torus Uniform connectivity

Anshul Kumar, CSE IITD slide 15 Illiac IV Mesh Network neighbors of node r : (r  1) mod 9 and (r  3) mod 9 Chordal Ring

Anshul Kumar, CSE IITD slide 16 Fat Tree Network

Anshul Kumar, CSE IITD slide 17 Dynamic Networks k  k cross -bar switch building block for multi-stage dynamic networks 2  2 switch straightexchangeupper broadcast lower broadcast simplest cross-bar

Anshul Kumar, CSE IITD slide 18 Baseline Network blocking can occur A B C A B C

Anshul Kumar, CSE IITD slide 19 Benes Network non-blocking

Anshul Kumar, CSE IITD slide 20 Network on Chip (NoC) Processing node Link Router

Anshul Kumar, CSE IITD slide 21 Switching Mechanism Circuit Switching (connection oriented communication) –A circuit is established between the source and the destination Packet Switching (connectionless communication) –Information is divided into packets and each packet is sent independently from node to node

Anshul Kumar, CSE IITD slide 22 Routing in Networks node incoming message outgoing message header payload/data store & forward routing worm hole routing time

Anshul Kumar, CSE IITD slide 23 Routing in presence of congestion Worm hole routing –When message header is blocked, many links get blocked with the message Solution: cut-through routing –When message header is blocked, tail is allowed to move, compressing the message into a single node

Anshul Kumar, CSE IITD slide 24 Routing Options Deterministic routing: always same path followed Adaptive routing: best path selected to minimize congestion Source based routing: message specifies path to destination Destination based routing: message specifies only destination address

Anshul Kumar, CSE IITD slide 25 Some Performance Parameters time sender receiver time of flight overhead Tx time=bytes/BW transport latency total latency

Anshul Kumar, CSE IITD slide 26 Other Parameters Throughput  Bandwidth (no credit for header) Bisection bandwidth = BW across a bisection Node degree Network Diameter Cost Fault Tolerance

Anshul Kumar, CSE IITD slide 27 Multidimensional Grid/Mesh Size =k  k  ….  k (n times) = k n Diameter = (k-1)  n without end around connections = k  n /2 with end around connections k-ary n-cube for (Binary) Hypercube : k = 2

Anshul Kumar, CSE IITD slide 28 Grid/Mesh Performance - 1 kdkd

Anshul Kumar, CSE IITD slide 29 Grid/Mesh Performance - 2

Anshul Kumar, CSE IITD slide 30 Switch Performance k  m cross -bar switch

Anshul Kumar, CSE IITD slide 31 Switch Performance – contd.

Anshul Kumar, CSE IITD slide 32 Switch Performance – contd.

Anshul Kumar, CSE IITD slide 33 Effect of re-submitted requests

Anshul Kumar, CSE IITD slide 34 BufferingBuffering Where are buffers located? Buffering before switching (k buffers, one at each input port) Buffering after switching (m buffers, one at each output port) Virtual output buffering (at each of the k inputs there are m buffers, one for each output)

Anshul Kumar, CSE IITD slide 35 BufferingBuffering k  m cross -bar switch k  m cross -bar switch k  m cross -bar switch B C, A C BA B CA C B, A C A B