1. Exploring Chip to Chip Photonic Networks
Philip Watts
Computer Laboratory
University of Cambridge
About myself: PhD and industrial experience in photonics and fibre-optic communications
Became interested in photonics for computer systems during work with Intel Research in
Fellowship for 2 years to study implications of latest photonic components for computer networks
Acknowledgement 1851
Will describe our proposed work
Acknowledgement: Royal Commission for the Exhibition of 1851

2. Exploring Chip-to-Chip Photonics
Bandwidth improvements in electronic interconnects have been achieved at the expense of increased latency and power consumption
Photonics has the potential to reduce power consumption/ latency
Known for a long time, BUT size, cost, manufacturing issues
Enabling technologies: silicon photonics and 3D integration
Needs a holistic, system assessment
assessing power consumption and performance

3. Photonics: Low Power and Latency ?
For point-to-point links:
Photonics power consumption is not dependent on distance, only on modulator/detector capacitance
Electrical interconnect power increases exponentially with distance
Photonics is not pin limited
Large bandwidth, low latency using by wavelength multiplexing
Switched systems:
Photonics switching power is not dependent on bandwidth
Electronic routers, power and latency consumed for each hop/buffer
10 mm best estimate of critical distance – 0.5 x chip with

4. Enabling Technology 1: Silicon Photonics
What’s changed?
Compact waveguides in Si/SiO2
Compact and low cost
Source: Intel
Off-chip, polymer waveguides
Integration on Copper PCBs
1.4m long spiral polymer waveguide with input from HeNe laser
Ring resonators can modulate, switch and filter
Switch/Modulate at >10Gb/s
Appropriate detectors and light-sources already exist
Diagram: M. Lipson (Cornell)

5. Enabling Technology 2: 3D Integration
Recent advances in 3D point toward multiple cores + DRAM layers (≈1 GB) + network within a single package
Combine such modules to produce larger systems for data centres or high performance computing
III-V substrates for light sources can be integrated
Chip-to-chip photonics gives a bigger win than on-chip, due to approx 10 mm critical distance
We will concentrate on chip-to-chip

6. Characteristics of a Photonic Network
Photonic Limitations:
Signals can not be (practically) buffered
Difficult to read header and setup switch on the fly
These limitations leads naturally to circuit switching (with edge buffers)
Photonics is good for flows/large packets
Relatively inefficient for small packets (e.g. cache lines)
Obvious network choice: Central Xbar/Clos switch with time slot access, e.g. SWIFT (Intel/Cambridge)
Various alternative distributed approaches proposed

7. FPGA-based Full System Emulation
FPGA-based emulation enables full system power/performance data using realistic workloads/timescales
100x faster than software simulation
Existing Computer Lab project, C3D, provides BEE3 infrastructure, cores and all-electronic reference design
Emulate core computer with photonic chip-to-chip network model
4 high-end FPGAs per board
4-8 MIPS-64 cores per FPGA
Clock rate ≈ 100 MHz
Levels of model abstraction
Parameterisable and synthesisable SystemVerilog network model
Photonic component power consumption model
Photonic path viability model
To discover apllications in which photonic networks give a big win, need full system emulation
Features of BEE3 board (UC Berkeley/Microsoft)

8. Exploring Chip-to-chip Photonics
Collaborators: Myself, Andrew Moore, Simon Moore (Cambridge), Robert Killey (UCL EE)
Program to investigate implications of chip-to-chip photonic interconnect on architecture of:
Data Centres
High Performance Computing
Build accurate and experimentally verified models of latest silicon photonics components (UCL)
Build full system FPGA-based emulator
Allows rapid network architectural exploration
Find applications which benefit from photonic networks
Determine power savings and performance gains and optimised
Find applications and network types which maximise power savings and performance gains

9. Backup

10. Distributed Switching
Columbia University Photonic NoC with Electronic Overlay
Cross-bar approach is limited by N2 scaling and complexity of arbitration/allocation
Example IBM/Corning OSMOSIS for 2000 ports/50m diameter
Alternatives using distributed switching
Mesh/Torus networks
path must be setup in advance => only suitable for large blocks of data
Wavelength encoded address
Packets dropped on contention => head of line blocking
Columbia University SpinNet

11. Ring Resonator Applications
4 port switch - switching multiple wavelengths simultaneously (e.g. 16 wavelengths x 10 Gb/s = 160 Gb/s)
Modulating multiple signals (1-20 GHz) onto multiple wavelengths on a single waveguide
Reverse process used to demultiplex wavelengths to recover individual signals
Diagrams: K.Bergman (Columbia), M.Lipson (Cornell)

12. 3D/SiP Technology
3D/SiP allows various processes including III-V substrates to be integrated in one package
Source: IBM

13. SOA Switching CAPE speciality
SOAs have amplification so no power budget issues
Can switch large bandwidth
High power
Require temperature stabilisation

14. Lockside Performance 4 x 4 mesh network
64 x 250 Mb/s = 16 Gb/s per link
Designed for on chip
Plenty of wiring resources available
For off-chip, each node would require 256 signal pins
Lightly loaded photonic cross-bar latency = 2 cycles (for 1-8 packet length)

15. Example: Photonic Cross-bar
Min slot time = (RTT to switch) + (arbitration time) ≈ 15 ns for 1m network radius
Switching time with ring resonators is v. low
Min latency = 2 slot times ≈ 30 ns
128b packet only uses 800ps!
15 ns also allows 280B per slot (16λ x 10Gb/s)
Currently creating initial model of chip-to-chip photonic network to add to C3D system
Compare with electronic VC router network
Initial design:
32 Nodes each serving a 32-core chip = 1000 cores
4 boards with 8 chips per board
Central x-bar switch, centralised routing and arbitration, fixed time slots
Network radius = 1m
Node bandwidth: 16 wavelengths at 5-10 Gb/s each = Gb/s

16. Multiple Networks
An idea that repeatedly comes up in literature: Two networks
One for large packets/flows
Photonic, circuit switched or with long slot
One for short packets, could be:
Photonic with minimum slot time
Electronic
Point2point photonic
Local (covering a single PCB) and global networks
Centralised cross-bar removes locality
Additional networks for broadcast/multicast
High bandwidth photonic signals can be split many ways without impedance matching issues
However, power budgets!
This is an area where Semiconductor Optical Amps may be useful

17. Photonic Interconnect
Some improvement in power and bandwidth/distance is obtained by simple replacement of electrical cables with fibre
We are interested in minimising power consumption using efficient switching and wavelength multiplexing
Interested in this ↓ not this ↓

18. Communication Centric Computer Design (C3D)
Computer architecture group project to investigate communications in multi-core systems
Implementation technology
Architecture
Algorithm mapping
Programming
Need accurate power consumption/performance data for full system
On-chip multi-core network using VC routers [Mullins 2006]

19. Silicon Photonics - Waveguides
Compact waveguides in Si/SiO2
400 x 200 nm typical
Bend radius ≈ few μm
Compact circuits
Low loss
CMOS compatible manufacturing
Low cost
Off-chip, polymer waveguides on PCB are in development
Source: Intel

20. Silicon Photonics – Active Devices
Ring resonators can modulate, switch and filter
3-50 μm diameter
Capacitances <5fF
Low power
Switch many wavelengths simultaneously
Modulation >10 Gb/s demo’ed
Ge photodetectors integrated with silicon waveguide
20 GHz bandwidth with high responsivity
Only need a light source
Can be off-chip
Diagram: M. Lipson (Cornell) d