1. ECE 1749H: Interconnection Networks for Parallel Computer Architectures: Flow Control Prof. Natalie Enright Jerger

2. Announcements Project Progress Reports – Due March 9, submit by e-mail – Worth 15% of project grade – 1 page Discuss current status of project Any difficulties/problems encountered Anticipated changes from original project proposal Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)2

3. Announcements (2) 2 presentations next week – Elastic-Buffer Flow Control for On-Chip Networks Presenter: Islam – Express Virtual Channels: Toward the ideal interconnection fabric Presenter: Yu 1 Critique due Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)3

4. Switching/Flow Control Overview Topology: determines connectivity of network Routing: determines paths through network Flow Control: determine allocation of resources to messages as they traverse network – Buffers and links – Significant impact on throughput and latency of network Winter 20114ECE 1749H: Interconnection Networks (Enright Jerger)

5. Flow Control Control state records – allocation of channels and buffers to packets – current state of packet traversing node Channel bandwidth advances flits from this node to next Buffers hold flits waiting for channel bandwidth Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)5 Buffer capacity Control State Channel Bandwidth

6. Packets Messages: composed of one or more packets – If message size is <= maximum packet size only one packet created Packets: composed of one or more flits Flit: flow control digit Phit: physical digit – Subdivides flit into chunks = to link width Winter 20116ECE 1749H: Interconnection Networks (Enright Jerger)

7. Packets (2) Off-chip: channel width limited by pins – Requires phits On-chip: abundant wiring means phit size == flit size RouteSeq# TypeVCID Packet Flit Head, Body, Tail, Head & Tail Phit HeaderPayload Head Flit Body FlitTail Flit Message Winter 20117ECE 1749H: Interconnection Networks (Enright Jerger)

8. Packets(3) Packet contains destination/route information – Flits may not  all flits of a packet must take same route RC Cache line TypeVCID Add r Head Flit Bytes 0-15Bytes 16-31Bytes 32-47Bytes 48-63 Body Flits Tail Flit RC Coherence Command TypeVCID Add r Head & Tail Flit Cmd Winter 20118ECE 1749H: Interconnection Networks (Enright Jerger)

9. Switching Different flow control techniques based on granularity – Message-based: allocation made at message granularity (circuit-switching) – Packet-based: allocation made to whole packets – Flit-based: allocation made on a flit-by-flit basis Winter 20119ECE 1749H: Interconnection Networks (Enright Jerger)

10. Message-Based Flow Control Coarsest granularity Circuit-switching – Pre-allocates resources across multiple hops Source to destination Resources = links Buffers are not necessary – Probe sent into network to reserve resources Winter 201110ECE 1749H: Interconnection Networks (Enright Jerger)

11. Circuit Switching Once probe sets up circuit – Message does not need to perform any routing or allocation at each network hop – Good for transferring large amounts of data Can amortize circuit setup cost by sending data with very low per- hop overheads No other message can use those resources until transfer is complete – Throughput can suffer due setup and hold time for circuits – Links are idle until setup is complete Winter 201111ECE 1749H: Interconnection Networks (Enright Jerger)

12. Circuit Switching Example Significant latency overhead prior to data transfer – Data transfer does not pay per-hop overhead for routing and allocation Acknowledgement Configuration Probe Data Circuit 0 5 Winter 201112ECE 1749H: Interconnection Networks (Enright Jerger)

13. Circuit Switching Example (2) When there is contention – Significant wait time – Message from 1  2 must wait Acknowledgement Configuration Probe Data Circuit 0 5 1 2 Winter 201113ECE 1749H: Interconnection Networks (Enright Jerger)

14. S 08 Location Time 0 1 2 5 8 S 08 A 08 D 08 T 08 S 28 01234567891011121314151617181920 S 28 Time to setup+ack circuit from 0 to 8 Time setup from 2 to 8 is blocked Time-Space Diagram: Circuit-Switching 012 345 678 Winter 201114ECE 1749H: Interconnection Networks (Enright Jerger)

15. Packet-based Flow Control Break messages into packets Interleave packets on links – Better utilization Requires per-node buffering to store in-flight packets Two types of packet-based techniques Winter 201115ECE 1749H: Interconnection Networks (Enright Jerger)

16. Store and Forward Links and buffers are allocated to entire packet Head flit waits at router until entire packet is received before being forwarded to the next hop Not suitable for on-chip – Requires buffering at each router to hold entire packet Packet cannot traverse link until buffering allocated to entire packet – Incurs high latencies (pays serialization latency at each hop) Winter 201116ECE 1749H: Interconnection Networks (Enright Jerger)

17. Store and Forward Example High per-hop latency – Serialization delay paid at each hop Larger buffering required 0 5 Total delay = 4 cycles per hop x 3 hops = 12 cycles Winter 201117ECE 1749H: Interconnection Networks (Enright Jerger)

18. H Location Time 0 1 2 5 8 01234567891011121314151617181920 BBBT 21 222324 HBBBT HBBBT HBBBT HBBBT Time-Space Diagram: Store and Forward Winter 201118ECE 1749H: Interconnection Networks (Enright Jerger)

19. Packet-based: Virtual Cut Through Links and Buffers allocated to entire packets Flits can proceed to next hop before tail flit has been received by current router – But only if next router has enough buffer space for entire packet Reduces the latency significantly compared to SAF But still requires large buffers – Unsuitable for on-chip Winter 201119ECE 1749H: Interconnection Networks (Enright Jerger)

20. Virtual Cut Through Example Lower per-hop latency Large buffering required 0 5 Total delay = 1 cycle per hop x 3 hops + serialization = 6 cycles Allocate 4 flit-sized buffers before head proceeds Winter 201120ECE 1749H: Interconnection Networks (Enright Jerger)

21. H Location Time 0 1 2 5 8 012345678 BBBT HBBBT HBBBT HBBBT HBBBT Time-Space Diagram: VCT Winter 201121ECE 1749H: Interconnection Networks (Enright Jerger)

22. Virtual Cut Through Throughput suffers from inefficient buffer allocation Cannot proceed because only 2 flit buffers available Winter 201122ECE 1749H: Interconnection Networks (Enright Jerger)

23. H Location Time 0 1 2 5 8 01234567891011 BBBT HBBBT HBBBT HBBBT HBBBT Insufficient Buffers Time-Space Diagram: VCT (2) Winter 201123ECE 1749H: Interconnection Networks (Enright Jerger)

24. Flit-Level Flow Control Help routers meet tight area/power constraints Flit can proceed to next router when there is buffer space available for that flit – Improves over SAF and VCT by allocating buffers on a flit-by-flit basis Winter 201124ECE 1749H: Interconnection Networks (Enright Jerger)

25. Wormhole Flow Control Pros – More efficient buffer utilization (good for on-chip) – Low latency Cons – Poor link utilization: if head flit becomes blocked, all links spanning length of packet are idle Cannot be re-allocated to different packet Suffers from head of line (HOL) blocking Winter 201125ECE 1749H: Interconnection Networks (Enright Jerger)

26. Wormhole Example 6 flit buffers/input port Blocked by other packets Channel idle but red packet blocked behind blue Buffer full: blue cannot proceed Red holds this channel: channel remains idle until read proceeds Winter 201126ECE 1749H: Interconnection Networks (Enright Jerger)

27. H Location Time 0 1 2 5 8 01234567891011 BBBT HBBBT HB HB HBBBT ContentionBBT BBT Time-Space Diagram: Wormhole Winter 201127ECE 1749H: Interconnection Networks (Enright Jerger)

28. Virtual Channels First proposed for deadlock avoidance – We’ll come back to this Can be applied to any flow control – First proposed with wormhole Winter 201128ECE 1749H: Interconnection Networks (Enright Jerger)

29. Virtual Channel Flow Control Virtual channels used to combat HOL blocking in wormhole Virtual channels: multiple flit queues per input port – Share same physical link (channel) Link utilization improved – Flits on different VC can pass blocked packet Winter 201129ECE 1749H: Interconnection Networks (Enright Jerger)

30. Virtual Channel Flow Control (2) Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)30 A (in) B (in) A (out) B (out)

31. Virtual Channel Flow Control (3) Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)31 AHA1A2A3A4A5 In1 BHB1B2B3B4B5 In2 AHBHA1B1A2 Out B2A3B3A4B4A5B5ATBT 112233 122333 3 AT 33 BT 3 3 3 2 21 1 3 2 2 11 A Downstream B Downstream AHA1A2A3A4A5AT BHB1B2B3B4B5BT

32. Virtual Channel Flow Control (3) Packets compete for VC on flit by flit basis In example: on downstream links, flits of each packet are available every other cycle Upstream links throttle because of limited buffers Does not mean links are idle – May be used by packet allocated to other VCs Winter 2011ECE 1749H: Interconnection Networks (Enright Jerger)32

33. Virtual Channel Example 6 flit buffers/input port 3 flit buffers/VC Blocked by other packets Buffer full: blue cannot proceed Winter 201133ECE 1749H: Interconnection Networks (Enright Jerger)

34. Summary of techniques LinksBuffersComments Circuit- Switching MessagesN/A (buffer-less)Setup & Ack Store and Forward Packet Head flit waits for tail Virtual Cut Through Packet Head can proceed WormholePacketFlitHOL Virtual Channel Flit Interleave flits of different packets Winter 201134ECE 1749H: Interconnection Networks (Enright Jerger)

35. Deadlock Using flow control to guarantee deadlock freedom give more flexible routing – Recall: routing restrictions needed for deadlock freedom If routing algorithm is not deadlock free – VCs can break resource cycle Each VC is time-multiplexed onto physical link – Holding VC implies holding associated buffer queue – Not tying up physical link resource Enforce order on VCs Winter 201135ECE 1749H: Interconnection Networks (Enright Jerger)

36. Deadlock: Enforce Order All message sent through VC 0 until cross dateline After dateline, assigned to VC 1 – Cannot be allocated to VC 0 again A0 A1 B0B1 C0 C1 D0D0 D1D1 Dateline Winter 201136ECE 1749H: Interconnection Networks (Enright Jerger) B B C C D D A A A A B B D D C C

37. Deadlock: Escape VCs Enforcing order lowers VC utilization – Previous example: VC 1 underutilized Escape Virtual Channels – Have 1 VC that is deadlock free – Example: VC 0 uses DOR, other VCs use arbitrary routing function – Access to VCs arbitrated fairly: packet always has chance of landing on escape VC Assign different message classes to different VCs to prevent protocol level deadlock – Prevent req-ack message cycles Winter 201137ECE 1749H: Interconnection Networks (Enright Jerger)

38. Buffer Backpressure Need mechanism to prevent buffer overflow – Avoid dropping packets – Upstream nodes need to know buffer availability at downstream routers Significant impact on throughput achieved by flow control Two common mechanisms – Credits – On-off Winter 201138ECE 1749H: Interconnection Networks (Enright Jerger)

39. Credit-Based Flow Control Upstream router stores credit counts for each downstream VC Upstream router – When flit forwarded Decrement credit count – Count == 0, buffer full, stop sending Downstream router – When flit forwarded and buffer freed Send credit to upstream router Upstream increments credit count Winter 201139ECE 1749H: Interconnection Networks (Enright Jerger)

40. Credit Timeline Round-trip credit delay: – Time between when buffer empties and when next flit can be processed from that buffer entry – If only single entry buffer, would result in significant throughput degradation – Important to size buffers to tolerate credit turn-around Node 1Node 2 Flit departs router t1 Credit Process t2 t3 Credit Flit Process t4 t5 Credit round trip delay Winter 201140ECE 1749H: Interconnection Networks (Enright Jerger)

41. On-Off Flow Control Credit: requires upstream signaling for every flit On-off: decreases upstream signaling Off signal – Sent when number of free buffers falls below threshold F off On signal – Sent when number of free buffers rises above threshold F on Winter 201141ECE 1749H: Interconnection Networks (Enright Jerger)

42. Process On-Off Timeline Less signaling but more buffering – On-chip buffers more expensive than wires Node 1Node 2 t1 Flit t2 F off threshold reached Off Flit On Process Flit t3 t4 t5 t6 t7 t8 F off set to prevent flits arriving before t4 from overflowing F on threshold reached F on set so that Node 2 does not run out of flits between t5 and t8 42

43. Buffer Utilization VA/ SA 1Cycle Credit count 2 Head Flit SA Body Flit 1 1 2 STLT Body Flit 2 Credit (head) BWVA/ SA ST CC-LTC- Up 000001 STLTBWSAST SASTLT CC-LTC- Up 2 Credit (body 1) SASTLT 3456789 10 1011 0 12 Tail Flit Winter 201143ECE 1749H: Interconnection Networks (Enright Jerger)

44. Buffer Sizing Prevent backpressure from limiting throughput – Buffers must hold flits >= turnaround time Assume: – 1 cycle propagation delay for data and credits – 1 cycle credit processing delay – 3 cycle router pipeline At least 6 flit buffers Winter 201144ECE 1749H: Interconnection Networks (Enright Jerger)

45. Actual Buffer Usage & Turnaround Delay Flit arrives at node 1 and uses buffer Flit leaves node 1 and credit is sent to node 0 Node 0 receives credit Node 0 processes credit, freed buffer reallocated to new flit New flit leaves Node 0 for Node 1 New flit arrives at Node 1 and reuses buffer Actual buffer usage Credit propagation delay Credit pipeline delay flit pipeline delay flit propagation delay Winter 201145ECE 1749H: Interconnection Networks (Enright Jerger) 113 1

46. Buffer Sizing: On/Off 9 cycles turnaround – Requires more buffering than credit-based Flit arrives at node 1 and uses buffers Off signal is sent Flit leaves Node 1. On signal sent to Node 0 Node 0 receives on signal Node 0 processes on signal. Buffer reallocated to new flit New flit leaves Node 0 New flit arrives at Node 1 and reuses buffer. Off signal sent On pipeline delay Flit pipeline delay Flit propagation delay Node 0 receives off signal On propagation delay Node 0 processes off signal and stops sending flits Flit pipeline delay, actual buffer usage Off propagation delay Off pipeline delay Winter 201146ECE 1749H: Interconnection Networks (Enright Jerger)

47. Flow Control and MPSoCs Wormhole flow control Real time performance requirements – Quality of Service – Guaranteed bandwidth allocated to each node Time division multiplexing Irregularity – Different buffer sizes Winter 201147ECE 1749H: Interconnection Networks (Enright Jerger)

48. Flow Control Summary On-chip networks require techniques with lower buffering requirements – Wormhole or Virtual Channel flow control Avoid dropping packets in on-chip environment – Requires buffer backpressure mechanism Complexity of flow control impacts router microarchitecture (in 2 weeks) Winter 201148ECE 1749H: Interconnection Networks (Enright Jerger)