1. Outer Tracker Off-Detector Readout and Control Discussion John Coughlan SLHC Tracker Readout Meeting March 7 th 2007

2. Starting Assumptions n Outer Tracker R > 60 cm u TOB &TEC (MSGCs) u Replace existing layers u Mini Strips n Readout of APV13 n Digital Readout n Just Readout (not Triggering) n Sparsification remains Off Detector ? n 50 nsec BX, 100 kHz L1 n Building blocks u Opto-Links 3-5 Gbps u Versatile Link GBT chipset n Based on material from Mark, Jan and Paulo from Feb Tracker Upgrade Workshop

3. Readout Channels n R > 60 cm 65% ~ 50 K APV25s ? n 10 Luminosity with 50 nsec BX -> 20 x track density n x 5 strips -> 4 x occ ~ 4% n ~ 50K APV25 -> 250 K APV13

4. Data Tracker Links n Readout FED & Control FEC decoupled n 40 K Uni Dir Pt2Pt links n Data ~ 80K APV25 n Synchronous System n Latency not critical FED APV

5. Mark : Further up the readout chain digital link interface functionality DAQ Function multiplexing encoding and fast serialization sparsification here maybe? off-detector rate = (no.of FE chips) x 160 Mbits/s digital header 128 analogue samples APV O/P Frame 20 Ms/s readout -> 7  s simple data rate / FE chip calculation current APV data frame duration 7  sec for 140 samples digitize at 8 bits -> 1120 bits to shift out in 7  sec => 160 Mbits/sec e.g. 20 FE chips on one link = 3.2 Gbits/s raw data (no sparsification / no encoding) if front end sparsification* (or faster links) then FE chips / link can increase Loss-less Compression ? 160 Mbits/sec (per chip – no sparsification) serial links 16 GBT

6. Readout Rates : Sparsification Off Detector n 3.2 Gbps link @ 80% payload ~ 2.6 Gbps data n APV13 128 ch -> 160 Mbps n 16 x APV13 per GBT / link = 15 K GBT / links n sFED : ORx 12 way x 8 ORx on 8U (feeding 8 x 12 MGBT FPGAs) n Input : 96 links/MGBT x 2.6 Gbps ~ 250 Gbps / sFED (cf FED ~ 25 Gbps) n Output (~4 x occ?) : / 10 -> 25 Gbps / sFED (cf FED 1.5 Gbps) Total System n ~ 250 K APV13 -> ~ 15 K links n ~ 160 sFEDs n ~ 10 sFED / ATCA crate -> 16 crates n ~ 300 Gbps / crate n ~ 2.5 Tbps to Filter Farm (cf FED 0.3 Tbps) Still a BIG system.

7. sFED Possible Implementation FPGA Switch Off Detector sFED SDRAM Buffer SNAP 12 Rear Transition Module STTC 10G Serial Backplane DAQ Crate Event Builder 12 x 3 Gbps APV13 ATCA Crate 8U Cntrl/Mon Power ORx FPGAs MGBTs GBT PHY MAC Sparsification 1 per ORx ORx and FPGA on Mezzanine? Keep FEC functions separate FED and FEC modules Communicate across backplane? ORx Mezzanine Prototype GBT

8. TTC Controls Tracker Links n 15 K Pt2Pt GBT links n Data ~ 250K APV135 n Non-Synchronous System n Latency not critical n 320 control rings n 2,500 Bi Dir n TTC + Control/Monitor EC n TTC Latency critical n Upgrade n 320 x 0.65 x 5 ~ 1,000 rings n 8 K links n Or bigger rings? Pt 2 Pt

9. Jan : Best of Both Worlds? n Broadcast downstream, Broadband upstream oRx DAQ FPGA oTx oRx eTRx oTx oRx oTx oRx oTx oRx oTx splitter TTC FPGA oRx protocol In detector FED FEC n Far Fewer Links. Downstream PON. Don’t need upstream PON n Every Data Link has dedicated TTC and Control. No more Rings. n  DAQ links no longer Synchronous (packets shared with EC mon) n  Couple Readout FED & Control FEC (EC cmds/mons) Coupling FE Module Pt 2 Pt PON

10. TTC Controls Tracker Links n 15 K Uni Dir n Data ~ 250K APV135 n No longer Synchronous n Data and Monitor EC n PON broadcast to 15K GBT n ‘FEC’ has OTx only n Each FE Hybrid has TTC/EC n No more rings n ‘CCU’ to chips via I2C PON Pt 2 Pt

11. Questions Simplex Pt2Pt Broadband for Data and Monitoring n How is this done? Protocols n Can we maintain data rates with protocols? I2C 16 x APVs FED FEC FPGAs

12. Questions n Simplex PON for TTC and Control n Splitting ratios? How many FEC links. n How to partition FED and FEC functions and inter communications? n What will FE modules look like? Assume Less Variations. n Do we need full FEC logic on Data Links (PD Off Detector?) n Consequences for Partitioning schemes? n What will new TTC system look like? n Assume New Emulators (keep in Global Trigger crate)

13. sFED/FEC Integrate FEC functions on FED? FPGA Switch Off Detector sFED SDRAM Buffer MPT 12 Rear Transition Module STTC Trigger Throttle 10G Serial Backplane DAQ Crate Event Builder 12 x 3 Gbps sTTC FPGAs MGBTs GBT Recv MAC ZS 1 per ORx APV13 Monitor ORx OTx ATCA Crate 8U Cntrl/Mon Ethernet Power ORx ORx and FPGA on Mezzanine?

14. Final System Ideas n New sDAQ (sFEDs connected direct to Filter via Super Event Builder Network) n New sTTC (Broadcasting Filter Addresses to FEDs) n Crates u Just Mechanics, Power, Cooling. -> Control/Monitoring via Ethernet. u Serial Backplane based crates (Telecom ATCA, VME46?). F Less Slots (but wider) F Better Power & Cooling ? F Better control & monitoring ? F FED Event Builder Crate

15. First steps to sFED n Year 1 : with FPGA Development boards n Implement GBT receiver functionalities in FPGA fabric (Verilog models?) n Channel density vs FPGA resources n Fast FPGA Memory interfaces n Implement protocols for extracting Data and EC Monitoring packets in FPGA n Year 2 : Prototype sFED on small form factor Mezzanine? n PCB Serial Integrity Issues ; Clock for FPGA MGBT Issues n Year 3 : Aim to instrument for full Tracker Readout chain test? n Learn from existing systems ; ECAL DCC, GCT, Tracker Trigger, other expts n Who else is looking at sFEDs? Common CMS solution? a la FEC n Get some experience with ATCA (other projects)

16. Other tasks n Front End Modularity? n sFEC module Trigger, Controls n Consider Sparsification On Detector ?

17. Spare Slides

18. Jan: Versatile Link: Definitions & GBT oTx oRx eTRxprotocol

19. Readout Rates : Sparsification On Detector n / 5? On Detector Total System n ~ 250 K APV13 n ~ 15 K -> ~ 3 K links n ~ 160 - > 80 sFEDs (limit of ATCA backplane switch) n ~ 8 crates n ~ 2.5 Tbps to Filter Farm (cf FED 0.3 Tbps)

20. Spare Slides n (Coexisting) Options: u User interface: Simple parallel bus u User interface: Industrial standard buses F Possible industrial standards: Ethernet PCIe (memory mapped bus) Hyper Transport (memory mapped) … Dedicated EC ASIC or “Client” ASIC

21. Control Links : Present System n 320 CCU rings x 2 (redundancy) DOH x 4 links n 2,560 links n Half Tx and Half Rx @ 40 Mbps? n ~ 40 FECs n 80K / 320 ~ 250 APV25 / CCU ring n 50 K APV25 -> 250 K APV13 n Keep sFEC implementation separate from sFED n Is there a need to broadcast additional TTC info to FEs?

22. CCU Ring APV25 DCU LD PLL MUX

23. Spare Slides

25. Kostas: Tracker Partition FED FEC FED crate FEC crate TTC ci TTC ex FE modules (xN) TTCoc Tracker Partition TTCoc FEC LTC APVE FMM Compact PCI crate TTC crate FRLs DAQ PC x 9

26. Kostas: Tracker Partition FED FEC FED crate FEC crate TTC ci TTC ex FE modules (xN) TTCoc Tracker Partition TTCoc FEC LTC APVE FMM Compact PCI crate TTC crate FRLs DAQ PC x 9 ?

27. CMS Optical Links web cms-opto:ecal-homecms-opto:ecal-home CMS TrackerCMS ECAL Digital Control Optical Link

28. Spare Slides FPGA