1. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 1 CMS Level-1 Upgrade Calorimeter Trigger Prototype Development P. Klabbers 1, M, Bachtis 1, J. Brooke 2, M. Cepeda Hermida 1, K. Compton 3, S. Dasu 1, A. Farmahni-Farahani 3, S. Fayer 4, R. Fobes 1, R. Frazier 2, C, Ghabrous, T. Gorski 1, A. Gregerson 3, G. Hall 4, C. Hunt 4, G. Iles 4, J. Jones 6, C. Lucas 2, R. Lucas 4, M. Magrans 5, D. Newbold 2,7, I. Ojalvo 1, A. Perugupalli 1, M. Pioppi 4, A.Rose 4, I. Ross 1, D. Sankey 7, M, Schulte 3, D. Seemuth 3, W.H. Smith 1, J. Tikalsky 1, A. Tapper 4, T. Williams 2 1 Physics Department, University of Wisconsin, Madison, WI, USA 2 University of Bristol, Bristol, UK 3 Engineering Department, University of Wisconsin, Madison, WI, USA 4 Imperial College, London, UK 5 CERN, Geneva, Switzerland 6 Iceberg Technology, UK 7 Rutherford Appleton Laboratory, UK TWEPP 2012 - September 18, 2012 The pdf file of this talk is available at: https://indico.cern.ch/contributionDisplay.py?contribId=86&sessionId=51&confId=170595

2. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 2 Present CMS Level-1 and Calorimeter Trigger Only calorimeter and muon systems participate e/ , jets, E T, H T, jet counts muons 3<|  |<5 |  |<3 |  |<3 |  |<1.6 0.9<|  |<2.4 |  |<1.2 4K 1.2 Gbaud serial links Cu cables Regional and Global Calorimeter Trigger (RCT and GCT) Pipelined system receives Trigger Primitives (TPs) from 8000 ECAL/HCAL/HF towers Finds 8 e/  candidates, creates 14 central tower sums, 28 quality bits, and forwards 8 HF towers and 8 HF quality bits

3. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 3 CMS Calorimeter Geometry EB, EE, HB, HE map to 18 RCT crates Provide e/  and jet,  E T triggers

4. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 4 Present Calorimeter Algorithms e/  Rank = Hit+Max Adjacent Tower Hit: H/E < Small Fraction Hit: 2 of 5-crystal strips >90% E T in 5x5 Tower (Fine Grain) Isolated e/  (3x3 Tower) Quiet neighbors: all 8 towers pass Fine Grain & H/E One of 4 corners 5 EM E T < Thr. Jet or  E T 12x12 trig. tower  E T sliding in 4x4 steps w/central 4x4 E T > others & > threshold  : isolated narrow energy deposits Energy spread outside  veto pattern sets veto  Jet  if all 9 4x4 region  vetoes off

5. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 5 Collisions at the LHC LHC currently delivering starting luminosities up to 7.5×10 33 /cm 2 s with 50ns bunch spacing at CMS and ATLAS Level-1 Trigger rates of 90 kHz Avg. number of interactions per crossing (pileup) ~30-35 to start LHC luminosity could increase up to 2×10 34 /cm 2 s by the end of 2017 (start of LHC long shutdown 2) 25 ns bunch spacing is the plan 50 ns may be easier and more reliable for the LHC Estimated average pileup from ~50 to >100 events per collision At CMS – Trigger and Detector Upgrades are essential to ensuring continued physics performance Keep thresholds as low as possible Reduce the effects of pileup Trigger can improve algorithms and resolution

6. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 6 Planned Improvements to Calorimeter Algorithms Electron/photon Use Hadronic Calorimeter depth segmentation ½ tower resolution Flexible isolation criteria, separate HCAL and ECAL Jet Improve resolution from 4 tower to 1 tower Use full Forward Calorimeter Granularity Flexible jet diameter 8-12 towers, circular or square Different algorithm options at the same time Tau Use smaller clusters, not 12x12 tower jets More candidates of each type Currently limited to 8 e/  of 2 types, 12 jets of 3 types (Central, Tau, Forward) Pileup Subtraction Move High-Level trigger PU corrections to Level-1 MET, HT, MHT Calculation Calculate E T Sums, Missing E T from clusters These all will improve resolution, rates, and efficiencies!

7. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 7 CMS Calorimeter Upgrade Level-1 Trigger Architecture Configurations Fully Pipelined Calorimeter Trigger Time Multiplexed Calorimeter Trigger Layer 1 Layer 2 Demux

8. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 8 Layers of the Calorimeter Trigger Layer 1 Reception of Trigger Primitives Option 1: Formation of trigger tower clusters and characterization bits Option 2: Multiplexing time slices of Trigger Primitives The filling Optical fibers Layer 2 Formation of Trigger Objects Pipelined – multiple processors and data sharing Time Multiplexed – processor/time slice, all algorithms in one processor

9. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 9 Layer 1

10. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 10 VadaTech V894 Crate Enhancement to “CMS Standard” VT892 Crate Supports 12 Double-Width, Full Height AMC Cards with redundant Power Supply and MCH Slots MCH1—commercial MCH module, used for GbE connectivity and IPMI control (part of AMC spec) MCH2—contains Boston University module, “AMC13”, for TTC downlink and crate DAQ interface Each AMC Slot Contains backplane 20 ports with a Tx & Rx pair Ports 0-3—for GbE, TTC, DAQ Ports 4-7—star fabric to slot MCH1 Ports 8-11—star fabric to slot MCH2 Port 12-15 and 17-20—not connected on VT892, but enhanced with custom fabric on VT894 A VT894 is otherwise identical to a VT892 with the addition of connections to otherwise unconnected ports 2 3 4 5 8 9 10 11 12 1 6 7 * *See backup slides for details

11. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 11 Wisconsin Calorimeter Trigger Processor (CTP) Virtex-6 Prototype Board Back End FPGA XC6VHX250T Front End FPGA XC6VHX250T Avago AFBR- 810B Tx Module 4X Avago AFBR- 820B Rx Module MMC Circuitry JTAG/USB Console Interface Mezzanine Power Supplies Dual SDRAM for dedicated DAQ and TCP/IP buffering

12. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 12 CTP-6 Features Dual Virtex-6 FPGAs (Front-End/Back-End), VHX250T or VHX380T with 6.5 Gbps-capable links Design optimized for the Compact Trigger Architecture FE FPGA has 48 Rx optical input links 24 intra-board links to forward data from the Front- End to Back-End FPGA 12 Backplane links (FE Tx, BE Rx) 12 Frontpanel Optical Outputs (from either FE or BE FPGA on per link basis) Supports TCP/IP for GbE connection Dedicated 25A power module for each FPGA logic core

13. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 13 CTP-6 Testing 2 CTP-6s Built Extensive loopback testing on the 12x outputs to the 48x inputs @ 6.4 Gbps over 5m cables Link drvr/rcvr settings affect results Have settings for error-free operation (see 98 hour test) Error free operation on the intra-card FE-to-BE links at 6.4 Gbps Currently surveying backplane links by moving CTP cards between different slots In process of firmware refit to make this more efficient About 25% of VT894 custom fabric links tested run @ 4.8 Gbps for ~30 min without errors

14. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 14 V894 Crate Test Setup BU AMC13 Vadatech MCH UW CTP- 6 TTC Downlink UW Aux

15. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 15 CTP-6 6.4 Gbps 98-hour Loopback 12x Loopback from BE FPGA to FE FPGA using 5m OM-3 cable 320MHz link clock derived from AMC13 40MHz Scope view at FE FPGA Rx pins Zero Bit Errors in the interval on all 12 fibers Pre/De-emphasis & equalization settings at Tx & Rx ports affect results

16. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 16 CTP-6 Summary New system of prototype hardware developed, including the CTP-6 and VadaTech 894 Backplane Not shown: Custom 25A power modules, Crosspoint I/O cards for data sharing currently being built Tests going well Validating the design formula Mechanical, Power, Balanced I/O Dedicated loopback tests with CTP-6 Surveying the VT894 backplane links A lot more to do Future designs in the pipeline CTP-7 as Xilinx 7-series products continue to become more readily available

17. Layer 2 CMS Calorimeter Trigger Upgrade - 17 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

18. MP7: Prototype Trigger Card P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London CMS Calorimeter Trigger Upgrade - 18

19. Extensive monitoring 15 voltage & current sensors 16 temperature sensors Firmware storage via MicroSD card or standard PROM MicroSD card allows fast storage of many firmware versions USB2 Console via microcontroller Virtex 7 based processing card Essentially large FPGA with a lot of I/O 1.0 - 1.4 Tb/s of optical I/O 48-72 Tx & 48-72 Rx @ 10G 50 Gb/s of electrical I/O 28 LVDS @ 1.8 Gb/s Dual 72Mb or 144Mb QDR RAM Clocked 500MHz The MP7 Summary CMS Calorimeter Trigger Upgrade - 19 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC- London

20. JTAG access to FPGA & microcontroller via Complex Programmable Logic Device (CPLD) verified QDR RAM functionality tested to 375Mhz 2x 13.5 Gb/s on each port Need to try @ 500MHz MMC code ported to MP7 from previous Mini-T* More monitoring than ever before All power supplies V/I/P, humidity, temperature, etc. IPbus** is stress tested 10 million packets, and no packets have been dropped. Test Status CMS Calorimeter Trigger Upgrade - 20 *See talk during TWEPP2011 **More details in this talk P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

21. Link Testing…. 240Gb/s (24 x10Gb/s) 240Gb/s (24 x10Gb/s) Tx Rx CMS Calorimeter Trigger Upgrade - 21 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

22. Simultaneous 48 channel 8B/10B encoding test Transmitted 7x10 13 bits per channel without any bit or alignment errors (includes data capture, counter and synchronisation). Simultaneous 24 channel PRBS31 (harsher) test with Xilinx IBERT Limited to ½ the channel due to IBERT software limitations Still valid because links split into 2 columns on each side of the die Transmitted 10 13 bits per channel without any errors Neither test had any special tuning (e.g. pre-emphasis) Tx & Rx of 1 Tb/s CMS Calorimeter Trigger Upgrade - 22 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

23. Clean optical eye (left) and electrical received eye (right). Still have to enable pre-emphasis in the optical receiver Should improve the rise time of the eye PCB manufacturing improvements are possible if needed. Preliminary SerDes results PRBS7 – Full Column (24 Chan) CMS Calorimeter Trigger Upgrade - 23 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

24. Card passes all basic tests JTAG chain based on CPLD OK FPGA OK All 48 Tx & 48 Rx channels running at 10G Transmitted ~ exabit during testing (10 18 ) without error QDR RAM operating (not yet tested at 500MHz) Microcontroller programmed and performing MMC duties Satisfied with card performance Will evaluate 144 link (rather than 92) @ 10G in November when the XC7VX690T becomes available. Summary MP7 CMS Calorimeter Trigger Upgrade - 24 P. Klabbers, U. Wisconsin, TWEPP September 2012, Slides from G. Iles, IC-London

25. Slides from Robert Frazier, Bristol University IPbus / μHAL The CMS experiment’s new hardware control “standard” Version 1.0 released in August 2012 Used by a growing number of other experiments also Hardware control via gigabit Ethernet UDP as the transport protocol (software support for TCP available) Complete solution is provided: IPbus/UDP Firmware module to add into your FPGA design µHAL application programming library ControlHub for serialising concurrent accesses from multiple clients Downloads + documentation: https://svnweb.cern.ch/trac/cactushttps://svnweb.cern.ch/trac/cactus 18th Sept 2012 TWEPP 2012 -- IPbus / μHAL, P Klabbers, U. Wisconsin 25

26. Slides from Robert Frazier, Bristol University IPbus is based on well-established networking technology Thus very flexible, with usage easily ranging from: IPbus / μHAL use-cases 18th Sept 2012 TWEPP 2012 -- IPbus / μHAL, P. Klabbers, U. Wisconsin 26 A single board on a bench......to something much bigger: IPbus Firmware footprint is small Real-world resource usage in a low-end Xilinx Spartan 6 (XC6LX16-CS324) FPGA ResourceUsage Registers7% Lookup Tables18% Block RAMs10%

27. Slides from Robert Frazier, Bristol University IPbus / μHAL Performance 18th Sept 2012 TWEPP 2012 -- IPbus / μHAL, P. Klabbers, U. Wisconsin 27 2 UDP packets required 3 UDP packets required Performance is dominated by latency. Current firmware only supports single UDP packet in flight per target device. To minimise network transports, requests are queued and only despatched when necessary. The next release of IPbus aims to improve performance figures furtherby: 1.Reducing firmware latency. 2.Support for multiple packets in flight. This should be available in early 2013.

28. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 28 SummarySummary Two FPGA-based high speed calorimeter trigger processing boards, and a new  TCA backplane built this year CTP-6 and VT894 Inter-crate sharing card, Crosspoint I/O being built MP7 Intense testing underway for both cards, backplane IPbus/  HAL tool available and in use These will make possible a CMS Level-1 calorimeter trigger upgrade for the LHC luminosity increases More sophisticated algorithms, resolution possible Can keep thresholds as low as possible to preserve physics Modularity will allow staging of new system to have slice ready by the end of LHC long shutdown 1 (end of 2014) and deploy the system in parallel Keep up with changing LHC conditions

29. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 29 Backup Slides

30. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 30 CMS Detector

31. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 31 Detector Front-ends Computing Services Readout Farms Event Manager Switch Fabric Level-1 Trigger Controls CMS Trigger & DAQ Systems Level-1 Trigger LHC beam crossing rate is 40 MHz & at full Luminosity of 10 34 cm -2 s -1  10 9 collisions/s Reduce to 100 kHz output to High Level Trigger and keep high-P T physics Pipelined at 40 MHz for dead time free operation Latency of only 3.2  sec for collection, decision, propagation

32. P. Klabbers, U. Wisconsin, TWEPP September 2012 CMS Calorimeter Trigger Upgrade - 32 V894 Custom Fabric by AMC Slot