Analysis of a Chip Multiprocessor Using Scientific Applications Gilbert Hendry Shoaib Kamil Marghoob Mohiyuddin Ankit Jain Leonid Oliker John Kubiatowicz John Shalf Aleksandr Biberman Johnnie Chan Benjamin G. Lee Luca P. Carloni Keren Bergman

Motivation CMPs of the future = 3D stacking Lots of data on chip Photonics offers key advantages Network layer Memory layers Multi-core processor layer International Symposium on Networks-on-Chip 11/11/2018

Why Photonics? OPTICS: ELECTRONICS: Photonics changes the rules for Bandwidth, Energy, and Distance. OPTICS: Modulate/receive high bandwidth data stream once per communication event. Broadband switch routes entire multi-wavelength stream. Off-chip BW = On-chip BW for nearly same power. ELECTRONICS: Buffer, receive and re-transmit at every router. Each bus lane routed independently. (P  NLANES) Off-chip BW is pin-limited and power hungry. TX RX RX RX RX RX RX TX RX TX TX TX TX TX TX TX TX International Symposium on Networks-on-Chip 11/11/2018

Silicon Photonic Integration Cornell, 2005 MIT, 2008 IBM, 2007 Sandia, 2008 Ghent, 2007 Columbia, 2008

Related Work Shacham, NOCS ‘07 Vantrease, ISCA ‘08 Batten, HOTI ‘08 International Symposium on Networks-on-Chip 11/11/2018

Hybrid Photonic Network Photonic Transmission Electronic Control Compute International Symposium on Networks-on-Chip 11/11/2018

Hybrid Photonic Network International Symposium on Networks-on-Chip 11/11/2018

Hybrid Photonic Network International Symposium on Networks-on-Chip 11/11/2018

Contributions This work achieves: Accurate simulation Application-based workloads Comparison of electronic and photonic networks International Symposium on Networks-on-Chip 11/11/2018

NanoPhotonic Devices Silicon waveguides Ring resonator (filter) Laser Ring resonator (modulator) Photodetectors Electronic data L. Chen, OE, 2008 International Symposium on Networks-on-Chip 11/11/2018

Switching Building Blocks Broadband 2×2 Switch B. G. Lee, ECOC 2008 Cross State Bar State  Transmission International Symposium on Networks-on-Chip 11/11/2018

Switch Characterization Broadband 1×2 Switch ER IL [A. Biberman et al., LEOS, 2007] Loss Parameter Value Waveguide propagation 0.5 dB/cm Waveguide crossing 0.05 dB Waveguide bend 0.005 dB/90o Passing by Micro-Ring (OFF) 0 dB Coupling into Micro-Ring (ON) 0.5 dB International Symposium on Networks-on-Chip 11/11/2018

Higher Order Switches N E S W International Symposium on Networks-on-Chip 11/11/2018

Simulation Environment Built in OMNeT++ Processing Plane Random, Trace, Execution Electronic Plane Routers XY routing Bubble Flow Control 4 VCs Pipelined – input, arbitration, output ORION – energy Circuit path setup logic Wires Custom lengths Photonic Plane Switches, modulators, detectors, filters, waveguides International Symposium on Networks-on-Chip 11/11/2018

Optical Loss Analysis Modulators Nonlinear effects Laser Switch Switch Total Injected power . PI = pi × Nλ Switch Injected power per wavelength Worst-case Insertion Loss Switch Received power Detectors Detector sensitivity Optical power International Symposium on Networks-on-Chip 11/11/2018

Insertion Loss Analysis International Symposium on Networks-on-Chip 11/11/2018

Experiment setup Networks Parameters Traffic Results International Symposium on Networks-on-Chip 11/11/2018

Experiment setup Networks Mesh Parameters Traffic Results Electronic Conc. Mesh Conc. Torus Electronic Mesh Concentrated Electronic Mesh Concentrated Electronic Torus International Symposium on Networks-on-Chip 11/11/2018

Experiment setup Networks Mesh Parameters Traffic Results Photonic Torus Networks Mesh Conc. Mesh Conc. Torus --------------- Ph. Torus Parameters Traffic Results International Symposium on Networks-on-Chip 11/11/2018

Selective Photonic Torus Experiment setup Parameters Traffic Results Networks Mesh Conc. Mesh Conc. Torus --------------- Ph. Torus Selective Selective Photonic Torus Ph. Torus El. Mesh Bandwidth 256B Msg size International Symposium on Networks-on-Chip 11/11/2018

Concentrated Photonic Torus Experiment setup Networks Mesh Conc. Mesh Conc. Torus --------------- Ph. Torus Selective Conc. Sel. Parameters Traffic Results Concentrated Photonic Torus Core Core Gateway Core Core International Symposium on Networks-on-Chip 11/11/2018

Simulation Parameters Networks Traffic Results Parameter Value Cores 64 Clock Frequency 5 GHz Data rate 10 Gb/s Parameters Network Channel Width Buffer Size (b) Electronic Mesh 128 1024 Conc. Electronic Mesh 2048 Conc. Electronic Torus Photonic Torus 32 512 Selective Photonic Torus 64 Conc. Photonic Torus Selective Conc. Ph. Tor. Energy Parameter Value PSE dynamic energy 375 fJ PSE static (OFF) power 400 µW Modulation dynamic energy 25 fJ/bit Modulation static power 30 µW Detector Energy 50 fJ/b Wire Energy ~50 fJ/bit/mm International Symposium on Networks-on-Chip 11/11/2018

Synthetic Benchmarks Networks Parameters Results Traffic Each transfer occurs 100 times Two versions: small (96B), large (128kB) Networks Parameters Results Traffic Bitreverse Random Neighbor Tornado International Symposium on Networks-on-Chip 11/11/2018

Scientific Applications Profiled by overloading communication functions in Linux Traces broken into phases to preserve order Random mapping Networks Parameters Results Traffic Gyrokinetic Toroidal Code (GTC) Cactus Application Num Phases Num Msgs Avg. Msg. Size (b) Cactus 2 285 25600 GTC 63 129796 MADbench 195 15414 5613 PARATEC 34 126059 43.3 PARAllel Total Energy Code (PARATEC) MADbench International Symposium on Networks-on-Chip 11/11/2018

Results – Synthetic (Small) Conc. Mesh Photonic Torus Conc. Torus Conc. Torus Selective Conc. Selective Electronic Photonic International Symposium on Networks-on-Chip 11/11/2018

Results – Synthetic (Large) Conc. Mesh Photonic Torus Conc. Torus Conc. Torus Selective Conc. Selective Electronic Photonic International Symposium on Networks-on-Chip 11/11/2018

Results - Applications Conc. Mesh Photonic Torus Conc. Torus Conc. Torus Selective Conc. Selective Electronic Photonic International Symposium on Networks-on-Chip 11/11/2018

Conclusions Detailed physically accurate simulations of future networks are informative. Photonics wins on energy consumption Significant difference in performance across different apps. Large messages/distances do well. Synergistic co-design of electronic and photonic planes may be beneficial International Symposium on Networks-on-Chip 11/11/2018