1. 1 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna DT Sector Collector Electronics Design and Construction Luigi Guiducci M. Dallavalle, A. Montanari, F. Odorici, G. Pellegrini, G. Torromeo, R. Travaglini Università di Bologna e INFN Sezione di Bologna TWEPP 2007 Prague, Czech Republic

2. 2 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Outline Introduction CMS muon detectors and DT Trigger overview DT Trigger Sector Collector main tasks and requirements Radiation issues and flash-based FPGAs choice DT Sector Collector hardware description Trigger data path Control and monitoring interfaces Main firmwares features Hardware production tests Screening tests Full I/O, 40 MHz complete tests Experience from commissioning Synchronization tools Spy data for trigger synchronization control

3. 3 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna CMS Drift Tubes Muon Detectors 2 Superlayers in the φ view, 1 in the θ view (SL = 4 planes of DTs) Maximum drift distance in the cell ~ 2 cm ≈ 400 ns = 16 BX 250 DT chambers in 4 concentrical stations, in 30° sectors, in 5 wheels Use of field in the yoke (~ 2T) for stand-alone Pt determination WHEEL (5) CHAMBER (250) CELL (~ 176k)

4. 4 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna DT Local Trigger Made of ~ 55000 ASICs Minicrates mounted on DTCs for both trigger and readout HW Trigger Outputs/BX: -2 φ-view track segments (bits): position (12), bending (10), reconstruction quality (3) - θ -view hits - 30 BX latency (16 BX drift time) Trigger links: 2 x FTP cables maximum length 40 m 8 LVDS links @ 480 Mb/s TS removes ghosts and select 2 “best” muons/BX TRACO correlates segments from the 2 φ SL BTI finds alignment of hits, assigns BX, φ, φ b

5. 5 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna DT Regional Trigger - Track Finder (DTTF) FPGA-based system in underground counting room Input: track segments from the detector via Opto links Extrapolation method to correlate track segments in the r-φ view and assign Pt from the bending Pattern matching method in the η view (straight tracks) For the DTTF, the input trigger data from all chambers must be synchronized at the same BX θ view Φ view

6. 6 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna LVDS OPTO Sector Collector: location and main tasks Located on detector towers, 60 SC boards (1/DT 30° sector), 2 VME 9U crates / wheel Main tasks: Synchronization of sector links Sorting of track segments candidates Monitoring of data Sector-level trigger generation and distribution Local trigger data sent on optical fibers to the underground counting room Sector Collector crates VME 9U with custom J3 backplane

7. 7 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Radiation tolerance In experimental cavern, outside the magnet yoke Neutron fluxes around CMS barrel are ~ 0.1—1 kHz/cm 2 for E > 100 keV Very low damage to electronics, but single event effects must be considered Memory cells can flip state  FPGAs configuration memory can be corrupted by SEE Experimental Hall Flash cells are more robust than SRAM cells (~15 V for programming, smaller) FLASH SRAM Near-detector towers: ProAsicPlus FPGA from Actel Configuration memory is FLASH-based Counting room : Altera StratixGX FPGA with 8 embedded gigabit transceivers Underground Counting Room

8. 8 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Implementation Input links Cat-5 FTP cables (480 Mb/s LVDS) from the minicrate Output links Optical fibers 9U motherboard : hosts RX/TX piggy boards, VME, TTC/Readout LVDS-RX mezzanines with 480 Mb/s deserializers and FPGA OptoTX mezzanine with 6 GOL chips at 1.6 Gb/s OptoRX boards (at DTTF crates) with deserializer FPGA Opto-RX VME TTC

9. 9 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna LVDS Receiver mezzanine Trigger data from one chamber @ 480 Mb/s Cable equalizers CLC014 Deserializers DS92LV1212 Actel ProAsic+300 BGA 456 balls Trigger data to OptoTX Spy and status data to board control device JTAG Interface (programming, spy, status, configuration) I2C interface (temperature sensor) Firmware main features : 4-to-1 segments sorting, I/O spy with 256 BX deep buffers (JTAG), test patterns generation, channel masking, output pipe for coarse delay adjustment

10. 10 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Trigger data paths Each LVDS receiver mezzanine delivers full trigger data to the Optical Transmitter mezzanine

11. 11 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Trigger data paths Spy Data Each LVDS receiver mezzanine delivers full trigger data to the Optical Transmitter mezzanine A partial copy of the trigger info (quality of the track segment, theta hits, bunch counters, parity checks, etc) is sent to the control chip at every bunch crossing

12. 12 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Trigger data paths Spy Data Each LVDS receiver mezzanine delivers full trigger data to the Optical Transmitter mezzanine A partial copy of the trigger info (quality of the track segment, theta hits, bunch counters, parity checks, etc) is sent to the control chip at every bunch crossing The control chip includes RAM- based buffers which can spy the track segments and their timing. These data can be accessed through VME or injected into the event data with a dedicated connection to the readout boards to READOUT VME Access

13. 13 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Local trigger generation Spy information can be used to generate a local trigger signal Logic function of triggers from the single chambers (coincidence, OR, anti-coincid...) These “local trigger” signals are sent in LVDS to sector chambers and to near ROS (DAQ) board In this way a local (sector-wide) trigger system is implemented

14. 14 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector – Jtag and I2C interfaces JTAG chain connects RX FPGAs Controlled through VME - Via Control Chip (1) - Via VME-JTAG bridge (2) Remote firmware upgrade Boundary scan Access to detailed RX status and control registers Access to RX spy buffers (full I/O x 256 BXs) allows a full minicrate trigger test I2C interface Controlled through VME Access to temperature / current / voltages sensors Access to GOL chips configuration and status (1) (2)

15. 15 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Optical Transmitter Mezzanine GOL chips @ 1.6 Gb/s Clock buffer NB100LVEP221FA QPLL & crystal VCSELs HFE4190-541 (850 nm) Trigger data from LVDS-RX (6x32 bit @ 40 MHz) Local trigger data 6 x 1.6 Gb/s I2C Interface (temperature, GOL bias current, laser monitoring) JTAG Interface (boudary scan)

16. 16 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Optical Receiver board Local trigger data @ 1.6 Gb/s Altera StratixGX – BGA 672 balls CMOS drivers (backplane fanout) To DTTF @ 40 MHz JTAG interface (config / status / boundary scan / FPGA programming) Firmware main features: Phi and Eta versions Embedded deserializers (4 or 5) 1.6 Gb/s  40 MHz Test patterns generation Per-channel output pipe for synchronization (up to 3 BXs in 6.25ns steps) Automatic check of test patterns sent by GOLs (allows > 10 11 patterns/hour) Custom JTAG chain for configuration, status (link unlocks, parity errors, etc) Amplifiers SY88883

17. 17 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Optical link monitoring and control GOL ADC TEMP VCSEL SY88883 RX FPGA I2C CTRL VME FPGA Control PC DTTF Board VME to JTAG Bridge Control PC PIN I2C JTAG Check amplifier threshold condition Check RX frequency lock Check RX data parities Check GOL test patterns Set GOL bias current Set/Unset GOL testmode Measure emitted light Measure temperature Software world Track Segments from Local Trigger Track segments to DT Track Finder TCP/IP

18. 18 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Tests/1 - Screening BOARDS: opto-tx lvds2ch Tested @ producer Piggy screening tests: Power & verify connectivity via JTAG boundary scan Static Patterns (on “inputs”) LVDS-TX JTAG interface PC to FPGA Static Patterns (on “outputs”) TEST BOARD FTP cables 480 Mbit/s Screening test at external firm for ~ 300 mezzanines Non-programmed FPGA  all I/Os are inputs Injecting static patterns on ALL I/Os and Boundary Scan access to FPGA Board inputs injected with serial LVDS at 480 Mbit/s  equalizers and deserializers are verified in this test !!

19. 19 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Test/2 – dynamic tests concept Pattern Unit is a VME testing device designed by CMS Bologna group: ● 128 I/O ● Up to 64k words ● Up to 100 MHz ● Multi board setup C++ software written to generate random data patterns, inject them into the board under test, and check the outputs - Verification of custom algorithms - Verification of all board components and connections

20. 20 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Tests/4 – SC production test bench SC under test TIM (TTC signals fanout) LVDS-TX connected to a Pattern Unit (data injection, emulates the minicrate) Custom J3 (TTC signals) Opto-rx connected to a Pattern Unit for pattern acquisition (emulates the DTTF) 6U VME crate (bridge, Patter Units, clock fanout...)

21. 21 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Tests/3 – SC production tests BOARDS: opto-tx lvds2ch opto-rx lvds4ch motherboard Tested @ producer Motherboard tests: Power, FPGA programming, VME, clocks, I2C, connectivity, SC to readout, JTAG chain Piggy onto Motherboard (mechanical assembling) Opto Tx-Rx chain test: GOL config. as “running counter”, check @ Rx end: locking, parity, sensitivity to VCSEL current, … LVDS-rx tests: FPGAs on board programming, dynamic tests with “test modes”, data phase scan, spy test with random patterns Full chain test with injected random patterns Piggy screening tests: Power & verify connectivity via JTAG boundary scan Tested @ INFN Bologna All blocks are fully automatic!

22. 22 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Commissioning During the last 5 months much work was done in the installation of tower electronics and connecting ▪Trigger ▪ Readout ▪ TTC ▪ DCS between the detector and USC55 After HW installation, a standard data taking procedure is followed on each sector with different runs ▪ Test Pulse ▪ Cosmics ▪ Noise 43/60 sectors commissioned Cosmics events from commissioning S10 S11 S12 Tower crates

23. 23 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Synchronization of the DT local trigger Test pulse procedure Pulses emulating vertical tracks are injected at the front-end This allows the integrity of the electronic chain – both trigger and readout – to be tested Trigger logic reconstructs local trigger segments at each pulse Test pulse can be used to check the synchronization of the chambers at the Sector Collector inputs

24. 24 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna RX mezzanine – input synchronization CKMB1 CKMB2 CKMB3 CKMB4 parity check parity error flag minicrates 30° sector of the DT detector either real data or test patterns Parity check in clock phase scan Each input channel has an independent, phase-adjustable clock. Parity checking ensures data integrity and it is used to evaluate synchronization parameters.

25. 25 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Output synchronization and spy CKMB? test pattern generator test mode enable trigger data CKMB? CKOUT SPY BUFFER RX CHIP # 1.. 4 Motherb. CONTROL CHIP trigger data to OptoTX VME CORE LOGIC RX mezzanines register data with a common clock at the outputs ( CKOUT here) Test patterns following the same path as real data can be generated Spy data are then used to check the synchronization between IN and OUT clock and the coarse synchronization between different chambers (BX-counters,...) After parameters tuning, real trigger data are spied and injected into the DAQ... DAQ CKDAQ dedicated test patterns

26. 26 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector synchronization Trigger synchronization can be checked looking at the spy data injected into the DAQ stream Example: number of triggers vs BX number for the four chambers in the sector One could check the composition of the entries...

27. 27 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Quality information... looking at the quality distribution at all BXs Note that MB4 has different configuration: only correlated triggers are admitted Other chambers show a large fraction of low quality uncorrelated triggers (3 hits) One could check the timing of segments with different qualities...

28. 28 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Timing of different quality segments... plotting the BX versus the track segment quality for the MB3 chamber of previous plot... Low quality uncorrelated segments (Li, Lo) composition is mainly ghosts and out of time tracks In commissioning, a single chamber track can generate a L1A (“technical trigger” ≠ DTTF) A cut on quality ≥ H is applied to each chamber, the resulting single chamber triggers are ORed

29. 29 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Conclusions The Sector Collector fulfils the I/O and timing requirements and is built with components suitable for operation in a moderate radiation environment FPGA designes embed several test and monitoring features; spy data can be accessed both via VME and by a fast connection to readout boards All hardware components were fully tested with custom HW & SW before integration Installation and commissioning provided lots of experience and did not point out specific weaknesses Tools for the synchronization of the local trigger primitives are being extensively used Event of DT-triggered cosmic muon seen by DT and ECAL (end of August CMS global run, Wheel0, underground cavern)

30. 30 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Zoom to event Φ - view θ - view

31. 31 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna END

32. 32 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Muon Detectors transverse view

33. 33 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna CMS “Cosmic Challenge” SC Crate YB+1: 1 SC board (sector 10) SC crate YB+2: 2 SC boards (sectors 10, 11) YB+1: Sector 10 MB1 – MB2 – MB3 YB+2: Sector 10 MB1 – MB2 – MB3 YB+2: Sector 11 MB1 – MB2 – MB3 Detector Tower DTTF: 3 PHTF – 1 WS – 1 BS BS Trigger into LTC/CTC  L1A Green barrack 2 x LVDS cables / chamber 3 (6) Opto fibers / sector L1A (fiber) L1A (copper) DT Trigger Setup at Cosmic Challenge

34. 34 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna SC synchronization @ MTCC /1: scope At the very beginning, we could check the synchronization of the segments at DTTF inputs... At a very low level! Non-null segments pulses (DTTF inputs) BS trigger output (DTTF built a track with quality > threshold) (J. Ero)

35. 35 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna SC synchronization @ MTCC /2: spy data plots MB1 MB2 MB3 WH+2 Sect10 WH+1 Sect10WH+2 Sect11 MB3 theta Plots from SC SPY theta hits outputted 7 BXs before phi segments MB3 timing is downgraded by a “special” configuration Timing was found to be stable across several days of running

36. 36 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Monitoring @ MTCC – trigger primitives quality “Default” configuration: Only correlated (HH+HL+LL) segments Here: HH missing on MB2 (most of 1 layer had HV off) “Pointing” configuration: Only uncorrelated (H I +L I ) phi segments out of MB3 - it is (H I +L I ) · H theta Plots from SC SPY

37. 37 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna DQM trigger plots @ commissioning – chamber detail 1 2 34 1 Track segment quality vs BX 2 First and second track segment qualities 3 First and second track segment BX 4 Theta hits BX

38. 38 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Eye pattern at 1.6 Gb/s 625 ps

39. 39 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Trigger system overview On-line reduction from interaction rate (~700 MHz) to storage rate (~ 100 Hz) Trigger chain based on 2 levels Level-1  custom processors  muon and calorimeters  pipelined (sync. 40 MHz)  no dead-time  3.2 μs overall latency High level  Software on CPU farm  ~ s latency Level-1 Accept? Storage

40. 40 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Level-1 Muon Trigger overview On-detector electronics  Counting room electronics  Underground Counting Room

41. 41 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Meantimer

42. 42 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Design Method

43. 43 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna Sector Collector Block Scheme

44. 44 TWEPP 2007, Prague - 07 SepL.Guiducci - INFN Bologna ER = - ln(1-CL)/N Cable transmission testsBER < 2 10 -12 Prototype boards testsBER < 2 10 -9 Production boards testsBER < 2 10 -6 Binomial distribution, with large N (number of patterns) and no errors detected: Bit error rates from tests