STS Readout chain – Revisited Readout-ASIC to ROC Interconnect Walter F.J. Müller, GSI, Darmstadt CBM Collaboration Meeting 13 th April 2010

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 2 Slides from FEE/DAQWorkshop

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 3 The CBM Generic Read-out Chain Detector Front-End Board Read-Out Controller Active Buffer Board FEBROC ABB Data Combiner Board DCB First Level Event Selector FLES TNet Data Control Data & Control Sync Data Digitization Custom ASIC FEE  DAQ Interface Local Pre- Processing Buffering System Synchronization DAQ  FLES Interface FPGA Coprocessor Event Selection CPU Farm DAQDetector Sub-SystemFLES

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 4 Options for CBM FLES Location CBM Service Building ~350m linear dist. Distance is preliminary!! 'Green Cube'

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 5 STS Hit Rates per 128 Channel Chip Hit rate [MHz] Sta 0 Sta 1 Sta 2 Sta 3 Sta 4 Sta 5 Sta 6 Sta 7 Total Total Very few chip in inner region have hit rate > 32 MHz >90% of chips have hit rates in MHz range Hit rate per chip Statistics 25 AGev 10 7 evt/sec

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 6 STS FEB Data Aggregation Cases FEBa8 FEBa4 FEBa2 FEBa1 2.5 Gbps serial link for FEB  ROC connect local chip  chip data interconnect

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 7 Proviso Proviso: link counts likely to go up to account for more background, overhead, head room, chips exceed 2.0 Gbps: 29 with ; 9 with # of FEBs and type distributions varies for stations # of links is quite similar for all stations STS FEB Type and Link Distribution FEB type Sta 0 Sta 1 Sta 2 Sta 3 Sta 4 Sta 5 Sta 6 Sta 7 tot FEBs tot links FEBa FEBa FEBa FEBa tot FEBs tot links FEB Type Statistics 25 AGev 10 7 evt/sec 40 bit message Allow 20% overhead/headroom

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 8 Where to do Opto Conversion ? Ansatz here:  keep FEB simple  FEB has Cu interfaces  do electrical-optical conversion in separate unit  this unit can act as a 'patch panel'  natural to work in bundles of 12 links... FEBa2 FEBa2 FEBa4 FEBa8 FEBa8 Cu patch cables 'active' patch panel Ribbon fiber cables

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 9 STS Quadrant 6 cm 5 cm STS-FEBs Likely not a good place for patch panels, faces the magnet coil directly Is this enough space for: copper links - 10 opto converters - 10 ribbon fiber cables plus other services like - power distribution - control & monitoring -,,,,,,, size of a post card

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 10 STS Interface Converter Board Proposal for STS interfacing  2m copper twinax x1, x2, x4 or x8  copper => optical on Postcard 12.7cm x 8.9cm  Conversion to optical signals with AOCs Does electrical and AOC connector fit on one Postcard ? Connector density analysis AOC connectors twinax connectors twinax connectors Passive Board Postcard miniHT connector Slide: U. Bruening

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 11 Proviso Proviso: link counts likely to go up to account for more background, overhead, head room,.... All STS data can be transported by ~280 ribbon cables Corresponds to 34 cm 2 cable cross section STS Link & Ribbon Cable Distribution Sta 0 Sta 1 Sta 2 Sta 3 Sta 4 Sta 5 Sta 6 Sta 7 total FEBs links cables/quadrant cables tot FEB&Cable Statistics 25 AGev 10 7 evt/sec 48 bit message

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 12 Proviso Proviso: link counts likely to go up to account for more background, overhead, head room,.... ROC/DCB STS Readout Chain in a Nutshell.... on/near Detector CBM Service Bldg. GSI/FAIR IT Bldg. ~50m~500m FLES chips 1512 FEBs 3300 links 280 cables 410 GB/sec 280 ROCs 1320 links 330 GB/sec 2.5 Gbps 5 Gbps

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 13 Physical Location View FEBROC/DCS ABBDCB FLES TNet Data Control Data & Control Sync Data Control STS,.... CLOSY TOF,...? ROC ECS Cu optical on/near DetectorCBM Service Bldg.GSI/FAIR IT Bldg. Data ~50m~500m ?

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 14 Towards a System Concept

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 15 Loose Ends So far only hit data considered  there are also clock, sync, and control connections So far we tried to avoid a 'support chip'  some data aggregation is needed on detector  so far 'distributed solutions' discussed Alternatives:  consider a 'support chip' acting as 'communication hub'  look again into the CERN GBT approach

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 16 GBT – Revisited

13 April 2010CBM Collaboration Meeting -- Walter F.J. Müller, GSI 17 GBT to front-end link topology  GBT to Front-end links:  32 (+1) bi-directional serial links (e-links)  E-link bandwidth is 80 Mb/s  Several e-links can be grouped together to serve a single front-end device achieving bandwidths that are multiples of 80 Mb/s  16-bits for TTC  8 phase adjustable clocks  An E-Link Port Adaptor (EPA) “macro” will be available for integration in the front-end ASICs Slide from: TWEPP-07 presentation of Paulo Moreira (see this link) link Shown in CBM DAQ on 24 th Jan 2008

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 18 Slide from: TWEPP-09 presentation of Paulo Moreira (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 19 Slide from: TWEPP-09 presentation of Paulo Moreira (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 20 Slide from: TWEPP-09 presentation of Paulo Moreira (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 21 Slide from: TWEPP-09 presentation of Paulo Moreira (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 22 CBM Profile

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 23 Profile of a 'communication hub' I notin RX-TX pairs much more TX links Tuned for very high data bandwidth – downstream side  optical data link usage not in RX-TX pairs  go for much more TX links than RX links  TX speed as high as feasible (e.g. 2.5 Gbps w/ 180nm)  RX speed doesn't need to be the same  handle multiple optical links with one 'hub' Tune for very high data bandwidth – readout-ASIC side  a single readout ASIC can produce 2.5 Gbps  go for 'simple to implement' interface  plausible choice: LVDS at 250 or better 500 Mbps The 'hub' must be able to aggregate data from several readout-ASICs onto a single output link. The 'hub' must handle data, clock, sync, and control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 24 Profile of a 'communication hub' II Ribbon cables with 12 fibers are widely used  standard opto converters are: 12TX, 12RX and 4TX-4RX  consider 12TX ribbon for data  consider 4RX – 4TX for clock, sync and control For a simple mapping of 'hub's to ribbons it is thus natural to consider 'hub's with  either 3, 6 or 12 TX for data  and 1RX-1TX for clock, sync and control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 25 Profile of a 'communication hub' III some assumptions Now some assumptions on what a feasible 'hub' can be: Assume a 'hub' ASIC with 6 fast serializers  6 TX for data  1 RX – 1 TX for clock, sync and control   15 Gbps data bandwitdth  2 'hub's will share a data ribbon  4 'hub's will share a control ribbon Assume 250 MHz system clock  500 Mbps LVDS seems plausible for readout-ASIC interface   support 5-8 times as many LVDS links as TX channels

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 26 Readout ASIC Interface xyter 6 fast LVDS data links 500 Mbps clock, sync, cntl interface clock, sync, cntl interface chainable in some form Cu optical

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 27 Readout ASIC Interface – FEB View feb as many LVDS links connected per chip as needed one clock, sync, cntl interface per FEB A high data rate FEB might have 6*8 LVDS links A low data rate FEB might have 1*8 LVDS links Cu optical

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 28 'Hub' ASIC Interface hub ~48 fast LVDS data links 500 Mbps ~6 clock, sync, cntl interface blocks Gbps data links TX control link pair Cu optical

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 29 Readout ASIC Interface – FEB View feb Low data rate FEB 1*8 LVDS links 1 hub servers several FEB hub Cu optical

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 30 Readout ASIC Interface – FEB View feb High data rate FEB 6*8 LVDS links 1 hub servers one FEB hub Cu optical

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 31 Opto-Converter Interfaces 4 RX – 4 TX for control links serves 4 hubs 12 TX for data links serves 2 hubs Cu optical opto Data Clock & Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 32 All Combined – On Detector Part feb Cu optical hubopto Data Clock & Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 33 All Combined – Detector & Service Bldg. Data Clock & Control CBM Service Bldg. DCB DCS on/near Detector Clock & Sync febhubopto TNet CntlNet Control Data

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 34 DCS DCB DCS DCB DCS Physical Location View - Revisited FEBABB DCB FLES Data Control Clock & Data & Control Clock & Sync Data STS,.... TOF,...? ROC ECS Cu optical on/near DetectorCBM Service Bldg.'Green Cube' Data ~50m~500m Control 'hub''opto' Data Clock & Control DCB DCS CLOSY DCS TNet CntlNet Clock & Sync Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 35 DCS DCB DCS DCB DCS Data Flow FEBABB DCB FLES Data Control Clock & Data & Control Clock & Sync Data STS,.... TOF,...? ROC ECS Cu optical on/near DetectorCBM Service Bldg.'Green Cube' Data ~50m~500m Control 'hub''opto' Data Clock & Control DCB DCS CLOSY DCS TNet CntlNet Clock & Sync Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 36 DCS DCB DCS DCB DCS Clock & Sync Flow FEBABB DCB FLES Data Control Clock & Data & Control Clock & Sync Data STS,.... TOF,...? ROC ECS Cu optical on/near DetectorCBM Service Bldg.'Green Cube' Data ~50m~500m Control 'hub''opto' Data Clock & Control DCB DCS CLOSY DCS TNet CntlNet Clock & Sync Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 37 DCS DCB DCS DCB DCS Control Flow FEBABB DCB FLES Data Control Clock & Data & Control Clock & Sync Data STS,.... TOF,...? ROC ECS Cu optical on/near DetectorCBM Service Bldg.'Green Cube' Data ~50m~500m Control 'hub''opto' Data Clock & Control DCB DCS CLOSY DCS TNet CntlNet Clock & Sync Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 38 Where to put the 'hub' ? STS view

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 39 Slide from: S. Igolkin, 3 rd Meeting of CBM-MPS STS consortium, June 2009 (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 40 Slide from: S. Igolkin, 3 rd Meeting of CBM-MPS STS consortium, June 2009 (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 41 Slide from: S. Igolkin, 3 rd Meeting of CBM-MPS STS consortium, June 2009 (see this link) link

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 42 STS Quadrant – A possible arrangement 6 cm 5 cm STS-FEBs STS-'hub'STS-'opto'

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 43 STS Mechanical & Connection Hierarchy Data Control CBM Service Bldg. DCB DCS Module febhubopto Super-ModuleStation Clock & Sync TNet CntlNet Control

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 44 Summary

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 45 Summary This is meant to start the discussion  no pre-mature conclusions at this point This is meant to start the discussion  no pre-mature conclusions at this point Decide on general architecture and parameters  back-end links number of links per 'hub' distribution of clock/sync/control/data on links  the GBT scheme with all-on-one bi-direction link pair is one choice  asymmetric schemes, using 12TX and 4TX-4RX ribbons is another  front-end links protocol for clock, sync, data and control connections signaling (LVDS,SLVS), encoding (8/10 or not), framing,.... Decide on required R&D  Building Blocks to be tested  serializer; driver & receiver cells;...

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 46 The End Thanks for your attention

13 April 2010 CBM Collaboration Meeting -- Walter F.J. Müller, GSI 47 Backups