1. Performance characteristics of COTS 10Gb/s Optical Links for SLHC Experiments Jan Troska, Markus Axer, Karl Gill, Robert Grabit, Raquel Macias Jareno, Etam Noah, Stefanos Dris, Francois Vasey CERN Jan Troska, Markus Axer, Karl Gill, Robert Grabit, Raquel Macias Jareno, Etam Noah, Stefanos Dris, Francois Vasey CERN

2. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch2 Overview  Introduction The need for speed COTS link speeds  10G link building blocks Data Aggregation XFP modules 10G COTS system overview  10G Link Demonstrator Preliminary Results  Summary & Conclusions

3. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch3 The Need for Speed (1)  Currently deployed systems in HEP Rad-hard detector to counting room –Digital: 40Mb/s - 1.6Gb/s oe.g. ATLAS, CMS, LHCb, ALICE, D0, CDF –Analogue: 40MS/s 8-9bit linear range oe.g. CMS COTS within counting rooms –Digital only, up to 3Gb/s in use  Currently implemented digital systems are partially based upon commercial standards e.g. 8b/10b Ethernet-type packets, G-link Only On-Detector parts are customised to meet stringent requirements

4. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch4 The Need for Speed (2)  Detectors are talking about needs for higher data transmission bandwidths e.g. SLHC upgrades to Trackers (more details in break-out meeting on SLHC links later today)  It would seem sensible to base such upgraded systems on commercial standards for higher speeds Interoperate with commercial parts in benign environment of the counting rooms Leverage test procedures & equipment developed by industry with much more effort than would be available in our community Using COTS if at all possible will reduce the overall cost of implementation on large scale in HEP experiments

5. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch5 Standard Optical Datalink Speeds  Standards for Telecoms and Datacoms up to the 10G regime Telecom HEP

6. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch6 Current GOL  How to translate data acquired at 40 or 80 MHz synchronously to higher data rates? 16 bits » 640 or 1280 Mb/s –800Mb/s or 1.6Gb/s with 8b/10b encoding 32 bits » 1.28 or 2.56 Gb/s (1.6 or 3.2Gb/s 8b/10b) Data Aggregation  XAUI standard allows for 4x 3.125Gb/s electrical to be muxed onto 1x 10Gb/s with 64b/66b re-encoding Base clock for this standard is very close to 160MHz, already commonly used in LHC systems

7. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch7 Anatomy of an XFP module  What appears to be a simple transceiver is actually quite complex - e.g. : Microcontroller Signal conditioning (bi-directional) Signal conditioning (bi-directional) Laser Driver XFI (Electrical i/o) XFI (Electrical i/o) LC-connectors (Optical i/o) LC-connectors (Optical i/o) TOSA (Transmitter Optical Sub-Assembly) TOSA (Transmitter Optical Sub-Assembly) ROSA, inc. TIA (Receiver OSA) ROSA, inc. TIA (Receiver OSA)

8. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch8 Inside an XFP (1)  TransImpedance Amplifier (TIA) Limiting amplifier for photodiode signals  Laser Driver Provides programmable bias current Provides programmable modulation current  Signal Conditioner Receive direction Signal Conditioner with Integrated Post Amplifier Transmit direction Signal conditioner equalizes long FR4 trace runs from Host ASIC/SerDes  Microcontroller Monitoring of chip statuses, modification of parameters, communication off-module c.f. Rx40 c.f. LLD, GOL c.f. Detector control systems

9. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch9 Inside an XFP (2)  Transmitter Optical Sub-Assembly (TOSA) VCSEL (850 or 1300nm) FP laser (1300nm) DFB laser (1300 or 1550nm, CWDM) Electro-absorption modulator (1550nm)  Receiver Optical Sub-Assembly (ROSA) PIN photodiode (850 or 1300nm) Avalanche Photodiode (1300 or 1550nm) c.f. current links

10. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch10 Processing & DAQ Processing & DAQ XFP Transceiver XFP Transceiver XFP Transceiver XFP Transceiver 10G COTS System  Building a 10G readout system from Commercial parts (as a starting point) Aggregation Digitization & Encoding Digitization & Encoding E.g. 4x 3.125 XAUI to 10GbE Detector

11. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch11 Towards a Link Demonstrator  Start with a COTS-based 10Gb/s link to understand what the issues will be when customizing for use within a Particle Physics Detector XFP Demonstration board –Optical data transmission properties XAUI-to-10G ASIC evaluation platform –Electrical interface issues –Data pattern dependencies

12. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch12 Lab Demonstrator  Installed an XFP evaluation platform in lab First performance measurements made on eye diagram XFP Module & Evaluation board XFP Module & Evaluation board PRBS Generator Optical Attenuator Optical Attenuator Sampling Oscilloscope Sampling Oscilloscope

13. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch13 Eye Diagram Measurements Total Jitter (p-p) = 36ps c.f. Bit Period (UI) = 100ps Electrical (XFI) Eye Mask

14. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch14 Eye Measurements (2)  Effect of attenuation increase Jitter & Noise (amplitude jitter) increase Eye Closes  Monitor “Loss of Lock” at Receiver Crude initial measurement of link margin Measured 17dB

15. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch15 Eye Measurements - Optical  Demo optical head lacked sensitivity Choice of instrumentation is very important  Observation of true pulse shapes important for optimization of laser - driver interface

16. 13 September 2005LECC 2005 Heidelberg - jan.troska@cern.ch16 Summary & Conclusions  COTS optical transceivers fall into two distinct categories, dictated by the speeds of standardised transmission protocols (GbE, FC, Telecom OC-xxx) Future use in HEP will be based to some extent on inter- operability with commercial parts, so have choice of two possible speed windows (<4Gb/s & 10Gb/s) to gain from existing systems experience  Experience gained from 10Gb/s lab demonstrator will be invaluable in assessing the performance of future high-speed link systems for deployment in HEP experiments Understand the performance metrics, and which parts of commercial components contribute to specific issues –E.g. power consumption, error performance under irradiation –COTS are a moving target, device improvements may well be to our advantage!