1. NoC: Network OR Chip? Israel Cidon Technion

2. Israel Cidon, Technion Technion’s NoC Research: PIs  Israel Cidon (networking)  Ran Ginosar (VLSI)  Idit Keidar (Dist. Systems)  Avinoam Kolodny (VLSI) Students:  Evgeny Bolotin,  Reuven Dobkin,  Zvika Guz,  Arkadiy Morgenshtein,  Zigi Walter  Roman Gindin

3. Israel Cidon, Technion Origins of the NoC concept Early publications:  Guerrier and Greiner (2000) – “ A generic architecture for on-chip packet-switched interconnections ”  Hemani, Jantsch, Kumar, Postula, Oberg,Millberg and Lindqvist (2000) – “Network on chip: An architecture for billion transistor era”  Dally and Towles (2001) – “Route packets, not wires: on-chip interconnection networks”  Wingard (2001) – “MicroNetwork-based integration of SoCs”  Rijpkema, Goossens and Wielage (2001) – “A router architecture for networks on silicon”  De Micheli and Benini (2002) – “Networks on chip: A new paradigm for systems on chip design”  Bolotin, Cidon Ginosar and Kolodny (2004) – “QNoC: QoS architecture and design process for network on chip”

4. Israel Cidon, Technion Evolution or Paradigm Shift? Computing module Network router Network link Architectural paradigm shift  Replace wire spaghetti by an intelligent network infrastructure Design paradigm shift  Busses and signals replaced by packets Organizational paradigm shift  Create a new discipline, a new infrastructure responsibility Bus

5. Israel Cidon, Technion Characteristics of a paradigm shift Addresses a critical and topical need Enables a quantum leap in productivity and application Resistance from legacy experts Requires a major change of mindset and skills! Think: Networking not Bus evolution! successful

6. Israel Cidon, Technion Critical needs addressed by NoC 3) Enable Chip Multi Processors 1) Efficient interconnect: delay, power, noise, scalability, reliability 2) Increase system integration productivity

7. Israel Cidon, Technion NoC offers Area and Power Scalability For Same Performance, compare the NoC: Simple Bus: Segmented Bus: Point-to Point: E. Bolotin at al., “Cost Considerations in Network on Chip”, Integration, special issue on Network on Chip, October 2004 Wire-areaand power:

8. Israel Cidon, Technion 4 Decades of Network 101 Evolved from busses and p-t-p connections Extensive architectures, modeling and analysis research Architecture is about optimizing network costs Different goals and element costs => different architectures:  Local Area Networks (LANs)  Metropolitan Area Networks (MANs)  System interconnect networks (SAN, InfiniBand …)  WAN (TCP/IP, ATM…)  Wireless networks Cross layered design Early architecture standardization is an optimization burden!

9. Israel Cidon, Technion 4 Decades of Network 101

10. Israel Cidon, Technion Local Area Networks (LANs) Critical need  Distributing operations and sharing of heterogeneous systems Constraints  Standardization Main Cost  Incremental cost (NICs, wiring) Typical optimized architecture:  Low cost hubs/switches  Tree like architecture  Exploit low cost local BW Shared media Broadcast  Host embedded NICs

11. Israel Cidon, Technion System interconnect (SAN, InfiniBand) Critical need  Create a powerful specialized system from low cost units Constraints  Low latency Main Cost  Total system cost per MIP Typical architecture:  Wormhole/cut through  Connection based  Over-provisioned network  High degree/regular topology  Specific optimizations (e.g. RDMA)

12. Israel Cidon, Technion WAN (TCP/IP, ATM…) Critical need  Global application networking (collaboration, WWW, file sharing, voice) Constraints  Scalability  Heterogeneous user and application QoS requirements Main Cost  Physical infrastructure (mainly long distance trunks) Typical architecture of choice:  Packet switching  Irregular, small degree networks of high speed trunks  Optimization of topology and link capacities

13. Israel Cidon, Technion CAN optimization The main cost(s)  Total Area  Power  Others Design time, verification and testability, The design envelope (constraints)  Collection of designs supported by a given chip  Convex hull of traffic requirements all configurations  QoS constraints  Other requirements (eg: design automation…) Optimization variables  Switching mechanism  QoS  Topology (incl. links capacities)  Routing  Flow and congestion control  Buffering  Application support …..

14. Israel Cidon, Technion One NoC does not fit all! Flexibility Reconfiguration rate single application General purpose computer at design time at boot time during run time ASIC CMP ASSP FPGA I. Cidon and K. Goossens, in “Networks on Chips”, G. De Micheli and L. Benini, Morgan Kaufmann, 2006

15. Israel Cidon, Technion One NoC does not fit all! A large solution range! Flexibility Traffic Unpredictability single application General purpose computer At design time At configuration Run time I. Cidon and K. Goossens, in “Networks on Chips”, G. De Micheli and L. Benini, Morgan Kaufmann, 2006 ASIC CMP ASSP FPGA

16. Israel Cidon, Technion Architecture of choice:  Wormhole or small frame switching  Small # of buffers, VCs, tables  Simple QoS mechanisms (which?)  Topology and routing optimized for cost Main cost  Power and area Design envelop / constraints  Well define inter-modules traffic  Automatic synthesis  Variable QoS requirement Apply paradigm to ASIC based NoC

17. Israel Cidon, Technion Example: QNoC Quality-of-service NoC architecture for ASICs Traffic requirements are known a-priori Overall approach  Wormhole switching  QoS based on priority classes  Small buffer/VC budget  In-order SP XY routing  Irregular topology  Optimized link capacities * E. Bolotin, I. Cidon, R. Ginosar and A. Kolodny., “QNoC: QoS architecture and design process for Network on Chip”, JSA special issue on NoC, 2004. (0,2) (0,0) (1,0) (0,3) (1,4) (0,4) (2,1) (2,0) (2,2)(2,3)(2,4) (4,3) (3,4) (4,4) R R RRR R RRRRR RR R R R R (5,0)

18. Israel Cidon, Technion Quality-of-Service in QNoC Multiple priority classes  Define latency  Preemptive  Possible ASIC classes Signaling Real Time Stream Read-Write DMA Block Transfer Statistical guarantees  E.g. <0.01% arrive later then required N T * E. Bolotin, I. Cidon, R. Ginosar and A. Kolodny., “QNoC: QoS architecture and design process for Network on Chip”, JSA special issue on NOC, 2004.

19. Israel Cidon, Technion Extract inter- module traffic QNoC Design Flow Place modules Allocate link capacities Verify QoS and cost

20. Israel Cidon, Technion Module QNoC Design Flow Extract inter- module traffic Place modules Allocate link capacities Verify QoS and cost R R RRR R RRRRR RR R R R RRR R RRRRR RR RRRR RR R RR R R R R

21. Israel Cidon, Technion Module R R RRR R RRRRR RR R R R R R R RRR R RRRRR RR R R R R Extract inter- module traffic Place modules Allocate link capacities Verify QoS and cost Optimize capacity for performance/power tradeoff Capacity allocation is a traditional WAN optimization problem, however: QNoC Design Flow

22. Israel Cidon, Technion Wormhole Delay Modeling Approximate delay analysis in wormhole networks  Multiple Virtual-Channels  Different link capacities  Different communication demands Flit interleaving delay approximation: Queuing delay: * I. Walter, Z. Guz, I. Cidon, R. Ginosar and A. Kolodny, “Efficient Link Capacity and QoS Design for Wormhole Network-on-Chip,” DATE 2006.

23. Israel Cidon, Technion The Capacity Allocation Problem Given:  system topology and routing  Each flow’s bandwidth (f i ) and delay bound (T i REQ ) Minimize total link capacity Such that:

24. Israel Cidon, Technion Capacity Allocation – Realistic Example A SoC-like system with realistic traffic demands and delay requirements “Classic” design: 41.8Gbit/sec Using the algorithm: 28.7Gbit/sec Total capacity reduced by 30% After optimization Before optimization 00010203 10111213 20212223

25. Israel Cidon, Technion Optimizing routing on Irregular Mesh Goal: Minimize the total size of routing tables Around the Block Dead End E. Bolotin, I. Cidon, R. Ginosar and A. Kolodny, "Routing Table Minimization for Irregular Mesh NoCs", DATE 2007.

26. Israel Cidon, Technion Saving Table Hardware Traditional solutions - full routing tables DDestination Based Routing - at router SSource Routing – at sources Solution idea:  Use Reduced Tables  Store only relevant destinations (PLA)  Default function (“Go XY” or “Don’t turn”) + Table for deviations

27. Israel Cidon, Technion Routing Heuristics for Irregular Mesh Distributed Routing (full tables) X-Y Routing with Deviation Tables Source Routing Source Routing for Deviation Points Systems with real applications Random problem instances

28. Israel Cidon, Technion Efficient Routing Results Scaling of Savings Savings Network Size

29. Israel Cidon, Technion NoC for Shared Memory CMP Constraints  Multiple access to coherent cache  Unpredictable traffic pattern  QoS requirements (fetch, pre-fetch) Main cost  CMP power / area per performance Architecture of choice:  Tailored for a given CMP  In-order/adaptive routing?  Simple QoS mechanisms?  Regular topology? is CMP symmetric?  Built in support functions (multicast, search…)

30. Israel Cidon, Technion NoC can facilitate critical transactions * E.Bolotin, Z. Guz, I.Cidon, R. Ginosar and A. Kolodny, “The Power of Priority: NoC based Distributed Cache Coherency”, NoCs 2007.

31. Israel Cidon, Technion Priority NoC: Results

32. Israel Cidon, Technion NoC Based FPGA Architecture Functional unit Routers NoC for inter- routing Configurable region – User logic Configurable network interface

33. Israel Cidon, Technion NoC for FPGA Design envelope / constraints  Many ASIC like applications for a given FPGA  Hard NoC infrastructure – efficient but inflexible  Soft logic is reusable but has inferior performance Average NoC cost of most demanding designs  Hard grid links and router logic  Total configured NoC Logic used Architecture of choice:  Regular and uniform grid  In-order/load balanced routing  Hard logic for links, routers  Soft logic for routing algorithms, headers, CNIs  Soft NoC tuning (routing, CNI) for a given implementation

34. Israel Cidon, Technion NoC Based FPGA Architecture Functional unit Routers NoC for inter- routing Configurable region – User logic Configurable network interface

35. Israel Cidon, Technion Source Toggle XY Unlike TXY, traffic to same destination is not split Maximum capacity similar to TXY The route is a bitwise XOR of source and destination ID Can be extended to weighted source toggle (WOT)

36. Israel Cidon, Technion Two Hotspots Maximum Capacity Design Envelope for various distances between the hotspots for WOT

37. Israel Cidon, Technion Generic NoC Problems Many shared problems across design spectrum, examples: Need for a low latency class of service Verification and predictability Power control of NoCs Centralized vs. distributed control Is single NoC enough per chip?  Bus examples suggest otherwise Hot modules slows incoming NoC traffic  Off chip systems  Shared memory subsystems  Expensive functional units

38. Israel Cidon, Technion IP3 Interface IP2 Interface IP1 Interface HM is not a local problem Transparent to NoC performance NoC clogging by hot modules Walter, Cidon, Ginosar and Kolodny, ”Access Regulation to Hot-Modules in Wormhole NoCs”, NOCS 2007.

39. Israel Cidon, Technion IP (HM) Interface No “fairness” is guarantied since routers’ arbitration is based on local state The further is the source from the destination, its worm has to win more arbitrations The HM module bandwidth isn’t fairly shared Source Fairness

40. Israel Cidon, Technion Hot Module Distributed Arbitration Control is distributed or centralized Centralized control can account for dependencies Requests and grants are sent at high service level Requests and grants includes additional data as needed  requested quota, source queue size, priority, deadline, etc.  Granted quota, scheduling of transmission's, etc. Initial credits hides light load request-grant latency

41. Israel Cidon, Technion Hot vs. non-Hot ModuleTraffic HM Traffic Without Control Other Traffic Without Control HM Traffic With Control Other Traffic With Control

42. Israel Cidon, Technion Conclusions NoC is a chip design paradigm shift Introduces many diverse and new networking challenges No killer NoC for all chips Should not comply with any X-AN concept  May include centralized mechanisms  May involve more than one NoC/Bus mechanisms  May combine several communication methodologies Low latency NoC/Bus for metadata and urgent signals Beware of early standardization and legacy barriers Mutual benefit for VLSI-Networking collaboration NoC: A Network AND A Chip