1. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 1 Trigger Upgrade Wesley H. Smith University of Wisconsin Darin Acosta University of Florida Sergo Jindariani Fermilab US CMS PMG, January 16, 2013

2. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 2 Performance and Schedule of the LHC Need to handle PU~50, and L  2E34 @ 13-14 TeV Effectively a factor of ~6 (or larger) increase in current L1 rates M.Lamont, CMS Week, also shown to Council, Dec.2012

3. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 3 Motivation: Trigger Rate Projections Trigger rate studies from special 8 TeV high-PU runs NB: Total L1 Trigger Rate < 100 kHz ! Linear with PU and lumi, but rates growing large Trig. > Thr.(GeV) 1 e/γ > 22 1 μ > 14 1 Jet > 128 Sum(Jets) > 150 2 e/γ > 22

4. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 4 Goals for CSC Trig. Upgrade - U. Florida, Rice, Texas A&M Remove limit of 3 segments per Muon Port Card Each Muon Port Card covers one sector Particularly problematic for collimated multi-muons w/rising occupancy Increase bandwidth in trigger links Occupancy of segments from chambers will exceed optical link bandwidth to CSC Track-Finder Sector Processors Dropped segments will degrade performance: lose momentum precision (higher rate) and/or tracks (inefficiency). Improve momentum resolution Make full use of all track information to approach best precision achievable for standalone muon reconstruction Steeper rate vs. p T threshold curve  increases safety margin to high luminosity and high pile-up Deliver higher precision on output track quantities to Global Trigger upgrade, & more μ candidates η ✕ϕ = 0.05 ✕ 2.5° → 0.0125 ✕ 0.015° Accommodate new algorithms like those in Higher Level Trigger invariant mass cuts, jet-lepton matching, … Possible seeding of future inner tracking trigger Requires new high bandwidth (optical) links to Global Trigger

5. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 5 EMU Trigger Upgrade CavernCounting Room μTCA: Advanced Mezzanine Cards from Telecommunications Computing Architecture (commercial telco, see backup slide)

6. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 6 Muon Port Card Upgrade Use the existing MPC main board Backplane interface to TMB remains unchanged 3 original optical links are still available New mezzanine card with new FPGA and new links x 60 + 20 spares

7. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 7 EMU Track-Finder Upgrade “Backplane Connector” “Sector Processor” x 12 Connection to VT892 standard backplane

8. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 8 EMU CSCTF Chassis Not US* *Processing the resulting tracks with RPC or CSC hits is a US responsibility

9. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 9 Muon Trigger M&S TOTAL: $701K Muon Port Cards: 80 x $1480 = $126,400 Optical Fibers & installation: $51,600 New estimate 4x as much under investigation CSC Track-Finder: $523,246 Module Preproduction (3): $63,630 Module Production (18): $357,480 uTCA chassis, optical parts, COTS: $102,136

10. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 10 Upgrade Cal. Trig. Algos. - U. Wisconsin Upgrade Cal. Trig. Algos. - U. Wisconsin Particle Cluster Finder Applies tower thresholds to Calorimeter Creates overlapped 2x2 clusters Cluster Overlap Filter Removes overlap between clusters Identifies local maxima Prunes low energy clusters Cluster Isolation and Particle ID Applied to local maxima Calculates isolation deposits around 2x2,2x3 clusters Identifies particles Jet reconstruction Applied on filtered clusters Groups clusters to jets Particle Sorter Sorts particles & outputs the most energetic ones MET,HT,MHT Calculation Calculates Et Sums, Missing Et from clusters New Technologies (μTCA, Links) enable the above All coded in Firmware & Tested (latency/resources)

11. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 11 Upgrade Cal. Trig. Performance Simulation work on stage-1 calorimeter trigger by FNAL, UI-Chicago, UC Davis, UC San Diego, U Wisconsin, MIT, Ohio State U. (subset of stage 2) Also exploring muon isolation

12. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 12 Calo Trigger Upgrade in Parallel: Split inputs from ECAL & HCAL HCAL energy ECAL energy Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF energy HCAL energy Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System oSLB oRM Install optical SLB and optical RM mezzanines during LS1 Install HCAL passive optical splitters during LS1 or YETS Install HCAL backend μHTR cards for input to new trigger Install HCAL frontend electronics after LS2 (finer segmentation)

13. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 13 Calorimeter Trigger Evolution 3 new calorimeter trigger μTCA crates  601 + spares Optical Receiver Modules 

14. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 14 Calorimeter Trigger Processor Virtex-6 Prototype Board (CTP-6) Back End FPGA XC6VHX250 T/ XC6VHX380 T Front End FPGA XC6VHX250 T/ XC6VHX380 T Avago AFBR- 810B Tx Module 4X Avago AFBR-820B Rx Module MMC Circuitry JTAG/USB Console Interface Mezzanine Power Modules Dual SDRAM for dedicated DAQ and TCP/IP buffering 12x Multi Gig Backplane Connections

15. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 15 Calorimeter Trigger Processor Virtex-7 (CTP-7) Replace 2 Virtex- 6s with a Virtex 7 for processing+ZYNQ for embedded TCP/IP endpoint 30A, 1V power module for FPGA logic core 3x CXP Pluggable modules for 36 Tx + 36 Rx 10G optical links 2x AFBR-820 modules for 24 Rx 10G optical links Simpler design to execute than the CTP-6 36 Total + spares Virtex-7 VX690T FPGA ZYNQ XC7Z030 EPP (optional) 1.5V Supply 2.5V Supply 3.3V Supply 1V 30A Supply CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx 12X Rx 12X Rx (CTP-6 CAD View)

16. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 16 CIO-X: crate interconnections (2/crate x 3 crates = 6 + spares) Controller (MMC and link mgmt) 4X Avago AFBR-79EQDZ QSFP+ Module Positions 4x4 Lane Bidirectiona l Multi Gig Backplane Connections Backplane Rx/Tx Redriver ICs (top and bottom sides)

17. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 17 VT894 Crate Test Setup (Final system: 3 crates w/ 12 CTP7 ea.) BU AMC13 Vadatech MCH UW CTP-6 TTC Downlink UW Aux

18. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 18 Final Calo Trigger Upgrade (“Stage 2”) Two modes of connectivity required Keep new trigger flexible in order to adapt to the needs of the evolving CMS physics program Both architectures have two processing layers Layer 1 optimized for backplane connectivity, Layer 2 for optical TMT architecture chosen as baseline Fully Pipelined Calorimeter Trigger Time Multiplexed Calorimeter Trigger Layer 1 Layer 2 Demux

19. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 19 Stage 2 Cal. Trig. Upgrade (TMT Architecture) US: UK:

20. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 20 Calorimeter Trigger M&S TOTAL: 1,149 K$ 42 CTP7s @ 17.5K$ each (36 needed plus 6 spares): 735 K$ FPGA: 12 K$ Optical: 3 K$ Board, Fabrication, Assembly, other 2.5 K$ 6 CTP7 prototypes @ 17.5 K$ each: 105 K$ 9 CIOx @ 2K$ each (6 needed plus 3 spares): 18 K$ Optical: 1.2 K$ Board, Fab Assembly, other 0.8 K$ 3 CIOx prototypes @ 2K$ each: 6 K$ 4 Vadatech μTCA crates : 40 K$ incl. MCH & PS (3 needed plus spare) @ 10 K$ 4 AMC13 Modules (3 needed plus spare) @ 5 K$: 15 K$ Optical Cables & Patch Panel between Layers 1 & 2: 20 K$ oRM’s: 700 (601 plus spares) @ 300 $: 210 K$.

21. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 21 Trigger Labor & Travel (non-physicist) Total Labor: $2.07M Muon Trigger Labor over 4 years: $ 840K MPC Electronic & Firmware Engineering: $160K CSCTF Electronic & Firmware Engineering: $480K Software Engineering: $200K Calorimeter Trigger Labor over 4 years: $ 1,230K Electronic Engineering: $ 380K Firmware Engineering: $ 530K Technician: $160K Software Engineering: $160K Travel over 4 years: $130K Muon: $70K Calorimeter: $60K NB: Resource Loading is not complete

22. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 22 Trigger WBS High Level Notes: Starts Nov. 1, 2013 Prototyping complete Not included Dictionary structure only Next: Schedule integration & Resources Needs alignment with CMS Structure Underway All WBS US only

23. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 23 Trigger WBS Detail

24. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 24 Schedule & Completion Trigger Schedule: Installation of components to provide parallel upgrade trigger installation/commissioning/operation during LS1. Installation/Commissioning of upgrade trigger system during 2015. Parallel Operation of Upgrade Trigger System at beginning of run in 2016 with old trigger system. Full sole operation of Upgrade Trigger System in 2017. Trigger Completion KPP: Demonstration of 99.9% (99.99% objective) agreement btw. upgrade trigger electronics & software emulation through test patterns Demonstration of reduction of trigger rates for electrons, photons, muons and taus for a reduction of less than 15% in efficiency Incorporation of unganged ME1/1a data into the endcap muon trigger logic.

25. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 25 RisksRisks MPC installation incomplete by end LS1 Consequence: unable to provide full inputs to upgrade CTCTF Mitigation: installation of remaining MPC mezzanines during 2015-2016 YETS. oSLB-oRM installation not complete by end LS1 Consequence: full parallel operation of final calorimeter trigger not possible until 2018 Mitigation: use parallel operation involving stage-1 calorimeter trigger hardware or use slice for validation Full HCAL μHTR system not commissioned by end 2016 Consequence: operation of final calorimeter trigger not possible during 2016 Mitigation: Continue to use stage-1 upgrade calorimeter trigger hardware which incorporates much stage-2 hardware.

26. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 26 BACKUPBACKUP

27. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 27 US Upg. Existing CMS Trigger & DAQ Overall Trigger & DAQ Architecture: 2 Levels: Level-1 Trigger: 25 ns input 3.2  s latency Design: Interaction rate: 1 GHz Bunch Crossing rate: 40 MHz 40 MHz x 25 PU = 1 GHz Level 1 Output: 100 kHz Output to Storage: 300-400 Hz Average Event Size: 0.5 MB Data production 1 TB/day UXC   USC Calorimeters : Muon Systems:

28. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 28 Motivation: Projected L1 Rates @ 2E34 Single e/γ trigger Black - 14 TeV MC (50 PU) Red - 8 TeV data (66 PU) Lepton triggers scale by ~2 for increased center of mass energy. Muons have poor control of rates at high thresholds H T (E T sum of jets) Black - 8 TeV data (66 PU) Red - 8 TeV data (45 PU) Rates shown for L inst =2×10 34 cm -2 s -1 Single muon trigger Black - 14 TeV MC (50 PU) Red - 8 TeV data (66 PU) Jet trigger rates are strongly dependent on PU Rate in kHz ! 100

29. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 29 Tools for upgrades: ATCA Advanced Telecommunications Computing Architecture ATCA Example: ATLAS Upgrade Calorimeter Trigger Topological Processor Card 12-chan. ribbon fiber optic modules Backpl. opt. ribbon fiber connector Example: μTCA derived from AMC std. used by CMS HCAL, Trig. Advanced Mezzanine Card Up to 12 AMC slots Processing modules 6 standard 10Gb/s point-to -point links from each slot to hub slots (more available) Redundant power, controls,clocks Each AMC can have in principle (20) 10 Gb/sec ports Backplane customization is routine & inexpensive

30. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 30 Cal. Trig. Stage 1 UK

31. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 31 Muon Trigger WBS 401.04.01Trigger Milestones & Interfaces401.04.03.05Endcap Muon Track-Finder 401.04.02Trigger Management401.04.03.05.01EMUTF Production Manage 401.04.03Muon Trigger401.04.03.05.02EMUTF Software 401.04.03.01Muon Trigger Milestones & Interfaces401.04.03.05.03EMUTF Firrmware 401.04.03.02Muon Trigger Management401.04.03.05.04EMUTF Components 401.04.03.03Muon Port Card Mezzanine401.04.03.05.04.01EMUTF Optics 401.04.03.03.01MPCM Production Manage401.04.03.05.04.02EMUTF FPGAs 401.04.03.03.02MPCM Software401.04.03.05.04.03EMUTF Misc. Components 401.04.03.03.03MPCM Firmware401.04.03.05.05EMUTF PCB Fabrication 401.04.03.03.04MPCM Components401.04.03.05.05.01EMUTF Processing Card Fab. 401.04.03.03.04.01MPCM Optics401.04.03.05.05.02EMUTF Optics Card Fab. 401.04.03.03.04.02MPCM FPGAs401.04.03.05.05.03EMUTF Memory Card Fab. 401.04.03.03.04.03MPCM Misc. Components401.04.03.05.05.04EMUTF Backplane Connector Fab. 401.04.03.03.05MPCM PCB Fabrication401.04.03.05.06EMUTF Assembly 401.04.03.03.06MPCM Assembly401.04.03.05.06.01EMUTF Processing Card Assy. 401.04.03.03.07MPCM Testing401.04.03.05.06.02EMUTF Optics Card Assy. 401.04.03.03.08MPCM Ship401.04.03.05.06.03EMUTF Memory Card Assy. 401.04.03.03.09MPCM on-site Testing401.04.03.05.06.04EMUTF Backplane Connector Assy. 401.04.03.03.10MPCM Installation401.04.03.05.07EMUTF Testing 401.04.03.03.11MPCM Comissioning401.04.03.05.08EMUTF Ship 401.04.03.04MPC-EMUTF Optical Fibers401.04.03.05.09EMUTF on-site Testing 401.04.03.04.01MPC-EMUTF OF Production Manage401.04.03.05.10EMUTF Installation 401.04.03.04.02MPC-EMUTF OF Components401.04.03.05.11EMUTF Comissioning 401.04.03.04.03MPC-EMUTF OF Fabrication401.04.03.06EMU TF Infrastructure 401.04.03.04.04MPC-EMUTF OF Testing401.04.03.06.01EMUTF Infrastructure Manage 401.04.03.04.05MPC-EMUTF OF Ship401.04.03.06.02EMUTF Vadatech Crates 401.04.03.04.06MPC-EMUTF OF Installation401.04.03.06.03EMUTF AMC13 401.04.03.04.07MPC-EMUTF OF Comissioning401.04.03.06.04EMUTF Optical Splitters & Patch Panel 401.04.03.06.05EMUTF Infrastucture Testing 401.04.03.06.06EMUTF Infrastructure Ship 401.04.03.06.07EMUTF Infrastructure on-site Testing 401.04.03.06.08EMUTF Infrastructure Installation 401.04.03.06.09EMUTF Infrastructure Comissioning

32. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 32 Calorimeter Trigger WBS 401.04.04Calorimeter Trigger401.04.04.05CTP Infrastructure 401.04.04.01Calorimeter Trigger Milestones & Interfaces401.04.04.05.01CTP Infrastructure Manage 401.04.04.02Calorimeter Trigger Management401.04.04.05.02CTP Vadatech Crates 401.04.04.03Calorimeter Trigger Processor401.04.04.05.03CTP AMC13 401.04.04.03.01CTP7 Production Manage401.04.04.05.04CTP Optical Cables & Patch Panel 401.04.04.03.02CTP7 Software401.04.04.05.05CTP Infrastucture Testing 401.04.04.03.03CTP7 Firmware401.04.04.05.06CTP Infrastructure Ship 401.04.04.03.04CTP7 Components401.04.04.05.07CTP Infrastructure on-site Testing 401.04.04.03.04.01CTP7 Optics401.04.04.05.08CTP Infrastructure Installation 401.04.04.03.04.02CTP7 FPGAs401.04.04.05.09CTP Infrastucture Comissioning 401.04.04.03.04.03CTP7 Misc. Components401.04.04.06Optical Receiver Mezzanines 401.04.04.03.05CTP7 PCB Fabrication401.04.04.06.01oRM Production Manage 401.04.04.03.06CTP7 Assembly401.04.04.06.02oRM Software 401.04.04.03.07CTP7 Testing401.04.04.06.03oRM Firmware 401.04.04.03.08CTP7 Ship401.04.04.06.04oRM Components 401.04.04.03.09CTP7 on-site Testing401.04.04.06.04.01oRM Optics 401.04.04.03.10CTP7 Installation401.04.04.06.04.02oRM FPGAs 401.04.04.03.11CTP7 Comissioning401.04.04.06.04.03oRM Misc. Components 401.04.04.04CIOx Card401.04.04.06.05oRM PCB Fabrication 401.04.04.04.01CIOx Production Manage401.04.04.06.06oRM Assembly 401.04.04.04.02CIOx Software401.04.04.06.07oRM Testing 401.04.04.04.03CIOx Firmware401.04.04.06.08oRM Ship 401.04.04.04.04CIOx Components401.04.04.06.09oRM on-site Testing 401.04.04.04.04.01CIOx Optics401.04.04.06.10oRM Installation 401.04.04.04.04.02CIOx FPGAs401.04.04.06.11oRM Comissioning 401.04.04.04.04.03CIOx Misc. Components401.04.04.07oSLB-oRM Optical Fibers 401.04.04.04.05CIOx PCB Fabrication401.04.04.07.01oSLB-oRM Production Manage 401.04.04.04.06CIOx Assembly401.04.04.07.02oSLB-oRM OF Components 401.04.04.04.07CIOx Testing401.04.04.07.03oSLB-oRM OF Fabrication 401.04.04.04.08CIOx Ship401.04.04.07.04oSLB-oRM OF Testing 401.04.04.04.09CIOx on-site Testing401.04.04.07.05oSLB-oRM OF Ship 401.04.04.04.10CIOx Installation401.04.04.07.06oSLB-oRM OF Installation 401.04.04.04.11CIOx Comissioning401.04.04.07.07oSLB-oRM OF Comissioning

33. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 33 Calorimeter Trigger Primitives Present Connections Regional Calorimeter Trigger To DAQ Via GCT HCAL HTR Cards To DAQ Via HCAL DCC2 Existing Copper Cables ECAL TCCs To DAQ Via ECAL DCC Existing Copper Cables

34. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 34 Calorimeter Trigger Primitives Connection Evolution Regional Calorimeter Trigger HCAL uHTR Cards ECAL TCCs SLHC Cal Trigger Processor Cards Optical Ribbons OSLBs ORMs Trigger Primitive Optical Patch Panel ECAL Opti. Ribbons ECAL Indiv. Fibers (LC) Optical Ribbons To DAQ Via BU “AMC13” To DAQ Via GCT To DAQ Via BU “AMC13” To DAQ Via ECAL DCC HCAL HTR Cards To DAQ Via HCAL DCC2 Existing Copper Cables HCAL Opti. Ribbons Optical Splitter Upstream of HCAL HTR/uHRT

35. Wesley Smith, U. Wisconsin, PMG, January 16, 2013 Trigger Upgrade – 35 Calorimeter Trigger Primitives Final Situation HCAL uHTR Cards ECAL TCCs SLHC Cal Trigger Processor Cards Optical Ribbons OSLBs ECAL Opti. Ribbons Optical Ribbons To DAQ Via BU “AMC13” To DAQ Via BU “AMC13” To DAQ Via ECAL DCC HCAL Opti. Ribbons Trigger Primitive Optical Patch Panel