Anshul Kumar, CSE IITD CSL718 : Multiprocessors Interconnection Mechanisms Performance Models 20 th April, 2006

Anshul Kumar, CSE IITD slide 2 Connecting Processors and Memories Shared Buses Interconnection Networks –Static Networks –Dynamic Networks PPPP MMM Interconnection Network M MMM PPPP MMM M MMM Global Interconnection Network MMM

Anshul Kumar, CSE IITD slide 3 Shared Bus each processor sees this picture: processing bus access prob of a processor using the bus =  prob of a processor not using the bus = 1 –  prob of none of the n processors using the bus = (1 –  ) n prob of at least one processor using the bus = 1 – (1 –  ) n achieved BW on a relative scale = 1 – (1 –  ) n required BW = n  available BW = 1

Anshul Kumar, CSE IITD slide 4 Effect of re-submitted requests AW  (1-P A ) 1-  +  P A 1-P A PAPA prob = q A prob = q W

Anshul Kumar, CSE IITD slide 7 Waiting time

Anshul Kumar, CSE IITD slide 8 Switched Networks BUS Shared media Lower Cost Lower throughput Scalability poor Switched Network Switched paths Higher cost Higher throughput Scalability better

Anshul Kumar, CSE IITD slide 9 Interconnection Networks Topology : who is connected to whom Direct / Indirect : where is switching done Static / Dynamic : when is switching done Circuit switching / packet switching : how are connections established Store & forward / worm hole routing : how is the path determined Centralized / distributed : how is switching controlled Synchronous/asynchronous : mode of operation

Anshul Kumar, CSE IITD slide 10 PMPM Direct and Indirect Networks PMSPMS PMSPMS SMPSMP SMPSMP PMPM PMPM PMPM SWITCH DIRECT INDIRECT node link node link

Anshul Kumar, CSE IITD slide 11 Static and Dynamic Networks Static Networks –fixed point to point connections –usually direct –each node pair may not have a direct connection –routing through nodes Dynamic Networks –connections established as per need –usually indirect –path can be established between any pair of nodes –routing through switches

Anshul Kumar, CSE IITD slide 12 Static Network Topologies Linear Star 2D-Mesh Tree Non-uniform connectivity

Anshul Kumar, CSE IITD slide 13 Static Networks Topologies- contd. Ring Fully Connected Torus Uniform connectivity

Anshul Kumar, CSE IITD slide 14 Illiac IV Mesh Network neighbors of node r : (r  1) mod 9 and (r  3) mod 9 Chordal Ring

Anshul Kumar, CSE IITD slide 15 Fat Tree Network

Anshul Kumar, CSE IITD slide 16 Dynamic Networks k  k cross -bar switch building block for multi-stage dynamic networks 2  2 switch straightexchangeupper broadcast lower broadcast simplest cross-bar

Anshul Kumar, CSE IITD slide 17 Baseline Network blocking can occur

Anshul Kumar, CSE IITD slide 18 Benes Network non-blocking

Anshul Kumar, CSE IITD slide 19 Switching Mechanism Circuit Switching (connection oriented communication) –A circuit is established between the source and the destination Packet Switching (connectionless communication) –Information is divided into packets and each packet is sent independently from node to node

Anshul Kumar, CSE IITD slide 20 Routing in Networks node incoming message outgoing message header payload/data store & forward routing worm hole routing time

Anshul Kumar, CSE IITD slide 21 Routing in presence of congestion Worm hole routing –When message header is blocked, many links get blocked with the message Solution: cut-through routing –When message header is blocked, tail is allowed to move, compressing the message into a single node

Anshul Kumar, CSE IITD slide 22 Routing Options Deterministic routing: always same path followed Adaptive routing: best path selected to minimize congestion Source based routing: message specifies path to destination Destination based routing: message specifies only destination address

Anshul Kumar, CSE IITD slide 23 Some Performance Parameters time sender receiver time of flight overhead Tx time=bytes/BW transport latency total latency

Anshul Kumar, CSE IITD slide 24 Other Parameters Throughput  Bandwidth (no credit for header) Bisection bandwidth = BW across a bisection Node degree Network Diameter Cost Fault Tolerance

Anshul Kumar, CSE IITD slide 25 Multidimensional Grid/Mesh Size =k  k  ….  k (n times) = k n Diameter = (k-1)  n without end around connections = k  n /2 with end around connections k-ary n-cube for (Binary) Hypercube : k = 2

Anshul Kumar, CSE IITD slide 26 Grid/Mesh Performance - 1 kdkd

Anshul Kumar, CSE IITD slide 27 Grid/Mesh Performance - 2

Anshul Kumar, CSE IITD slide 28 Grid/Mesh Performance - 3 k-ary n-cube

Anshul Kumar, CSE IITD slide 29 Switch Performance k  m cross -bar switch

Anshul Kumar, CSE IITD slide 30 Switch Performance – contd.

Anshul Kumar, CSE IITD slide 31 Switch Performance – contd.

Anshul Kumar, CSE IITD slide 32 Effect of re-submitted requests

Anshul Kumar, CSE IITD slide 33 Effect of buffering There are two possibilities Buffering before switching (k buffers, one at each input port) Buffering after switching (m buffers, one at each output port)

Anshul Kumar, CSE IITD slide 34 Switch with input buffers Rate of messages at input and output of each queue is same in steady state - r per cycle Service time includes delays due to conflicts, calculated as earlier. This has an exponential distribution – recall the analysis for a shared bus. M/M/1 open queue model can be used to calculate queuing delay. Details are omitted.

Anshul Kumar, CSE IITD slide 35 Switch with output buffers Here we assume that all the messages destined for same output are queued in the same buffer, in some order. That is no rejections and no re-submissions. For each queue, Messages arriving per service cycle =  = Prob of a request coming from one of the k sources = p = Apply M B /D/1 model for finding queuing delay T w

Anshul Kumar, CSE IITD slide 36 ReferencesReferences D. Sima, T. Fountain, P. Kacsuk, "Advanced Computer Architectures : A Design Space Approach", Addison Wesley, K. Hwang, "Advanced Computer Architecture : Parallelism, Scalability, Programmability", McGraw Hill, 1993.