1. Consideration of the LAr LDPS for the MM Trigger Processor Kenneth Johns University of Arizona Block diagrams and some slides are edited from those of Guy Perrot (LAPP)

2. The LAr Calorimeter upgrade reduces the L1 EM object trigger rates by using finer granularity information (SuperCells versus Towers) as input to the L1 Calo Trigger system – Exploits differences in shower shape between signal and background Data flow – SuperCell signals are digitized on-detector at 40 MHz (320 SC per LTDB (LAr Trigger Digitizer Board) – Data is optically transmitted to the Back-end electronics at 5.12 Gbps (using the LOCx2 and LOCld) – There, the ADC data is converted to energy and time and sent to the L1 Calo Trigger System and TDAQ using the LDPS (LAr Digital Processing System) Overview 2

3. LTDB = ADDC – LAr Trigger Digitizer Board = ART Data Driver Card LOCx2+LOCld = GBT LDPS = MM Trigger Processor – LAr Digital Processing System = MM Trigger Processor LAr to Muon Dictionary

4. MM Trigger Processor 4 ADDC (ART Data Driver Cards) ART Data @ 40 MHz MM Trigger Processor Blades L1 Muon Sector Logic @ 40 MHz GBT Links ADDC configuration and monitoring, TTC TTC Partition Partition Master PC ATCA System Manager 10/40GbE via fabric TTC ROS PCs TDAQ To HLT Data Monitoring PCs DCS FELIX To shelf managers and Base interface (GbE) FE FELIX TTC DCS GBT Links Via Zone 3 Busy ADDC Config & Monitoring BE Configuration & Monitoring MM Trigger Processor programming, configuration, monitoring

5. LAr LDPS blade – Input: 40 (48) x 4 optical fibers at 5.12 Gbps – Output: 40 (48) x 4 optical fibers at ~10+ Gbps to L1 Calo – Output: Input data and results to TDAQ – Output: Input data and results to private Monitor PCs MM Trigger blade – Input: 32 (36) x 4 optical fibers at 4.48 Gbps (Wide Bus mode) – Output: 1 x 4 optical fibers at 4.48? Gbps – Output: Input data and results to TDAQ – Output: ?? Input data and results to private Monitor PCs LAr and Muon Backend I/O

6. ADDC to AMC data flow AMC to Muon Sector Logic data flow GbE Base Interface – Board Configuration – Board Monitoring – Histogramming 10GbE Fabric interface – Data monitoring (Fast) GBT links – TTC/Busy – TDAQ Data JTAG IPM Bus – Only ATCA management! Power MM Trigger Processor Communications 6

7. MM Trigger Processor 7 AMC 1/10 GbE Switch Zone 2 : Base Zone 1 Zone 2 : Fabric Zone 3: RTM Ch1 1GbE Ch2 1GbE ATCA Power 400W IPMC 1 GbE TTC 1 GbE TTC 1 GbE TTC 1 GbE TTC XAUI 10Gbps (PCIe?) ATCA Switches (AMC Reprogramming, AMC Configuration, AMC Monitoring) ATCA Shelf Manager I2C IPMB Ch2 10GbE Ch1 10GbE 4 4 1 1 4 FPGA 16 1 1 4 4 4 4 1 4 4 4 4 RJ45 IPML,JTAG,Ref Clks… XAUI (PCIe?) 1 GbE IPML, JTAG, I2C Sensors I2C GbE (Debugging) TTC µ POD Tx (x4) µ POD Rx (x4) 1 Tx links µ POD Tx (x4) µ POD Rx (x4) µ POD Tx (x4) µ POD Rx (x4) µ POD Tx (x4) µ POD Rx (x4) FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 4 Up to 4 GBT connectors 1 1 1 FELIX GBT GbE ATCA Shelf Power 48V Power (Data Monitoring) ATCA Switches (ATLAS Event-TDAQ) (TTC) 10/100MbE 32 Rx links 1 Tx links 32 Rx links 1 Tx links 32 Rx links 1 Tx links 32 Rx links GBT 1GbE (ATCA Management, Monitoring) MMC 4 GBT 10GbE 1 1 1 XAUI 10Gbps (PCIe?) 1 4 GBT 1 GbE JTAG I2C IPML,JTAG,Ref Clks… Ref Clks (TTC, XAUI…) 1 1 JTAG Muon Sector Logic Data @ 40 MHz Muon Sector Logic Data @ 40 MHz Muon Sector Logic Data @ 40 MHz Muon Sector Logic Data @ 40 MHz ADDC Data @ 40 MHz ADDC Data @ 40 MHz ADDC Data @ 40 MHz ADDC Data @ 40 MHz

8. ADDC to AMC data flow AMC to Muon Sector Logic data flow GbE Base Interface – Board Configuration – Board Monitoring – Histogramming 10GbE Fabric interface – Data monitoring (Fast) GBT links – TTC/Busy – TDAQ Data JTAG IPM Bus – Only ATCA management! Power MM Trigger Processor Communications 8

9. Design (Stony Brook) and layout (BNL) are complete Fabricated AMC board expected before 12/1 All parts in hand except MicroPOD optics, optical cables, 40 MHz clock and front panels Firmware support (Arizona) in progress – Perhaps available to users May 2014 Longer term, LAPP holds responsibility for ATLAS AMC design and production – Unclear if LAPP design will follow AMC Demonstrator design AMC Demonstrator Status

10. Carrier demonstrator cards (including IPMC) with Altera FPGA have been produced by LAPP and will be made available to Stony Brook and Arizona soon LAPP Carrier demonstrator card design currently being translated into Stony Brook PCB design software Stony Brook will design a second Carrier card demonstrator with a GbE switch and increased power for AMC slots – Perhaps available May 2014 Longer term, Stony Brook, BNL, Arizona hold responsibility for ATLAS Carrier design, production and firmware AMC Carrier Card Status

11. 11 AMC Parts Layout (top ) Xilinx Vertex-7 4 micro-POD arrays 2x12 receivers and 2x12 transmitters each DDR3 RAM chips

12. 12 AMC Parts Layout (bottom) Clock chips

13. 16 layers total with thru vias Material N4000-13EP AMC Routing (top layer) 13

14. The LAr LDPS is a developed candidate for the MM Trigger Processor – Pros Takes advantage of an existing R&D program to develop AMC cards using high density Avago MicroPODs and FPGAs for Serdes and supporting Carrier cards Though not final, data paths are defined for TDAQ, TTC, configuration, monitoring, etc. AMC and Carrier card demonstrators well along and both AMC and Carrier cards will be available for users by May 2014 – Cons Output fibers for Muon Sector Logic << Output fibers for L1 Calo trigger Monitor stream (10 GbE out) probably overkill Unknown compatibility with sTGC Trigger Processor Conclusions 14

15. Backup 15

16. LAr LDPS 16 LTDBs ADC Data @ 40 MHz LDPBs L1 Calo Data @ 40 MHz GBT Links LTDB Configuration & Monitoring, TTC TTC Partition Partition master PC ATCA System Manager 10/40GbE via fabric TTC ROS PCs TDAQ To HLT Data Monitoring PCs DCS FELIX To shelf managers and Base interface (GbE) FE FELIX TTC DCS GBT Links Via Zone 3 Busy LTDB Configuration & Monitoring BE Configuration & Monitoring LDPB programming, configuration, monitoring

17. LDPB Block Diagram 17 (e/jFEX Data@40MHz) AMC 1/10 GbE Switch Zone 2 : Base Zone 1 Zone 2 : Fabric Zone 3: RTM Ch1 1GbE Ch2 1GbE ATCA Power 400W IPMC 1 GbE TTC 1 GbE TTC 1 GbE TTC 1 GbE TTC XAUI 10Gbps (PCIe?) ATCA Switches (LDPB Reprogramming, LDPB Configuration, LDPB Monitoring) ATCA Shelf Manager I2C IPMB Ch2 10GbE Ch1 10GbE 4 4 1 1 4 FPGA 16 1 1 4 4 4 4 1 4 4 4 4 RJ45 IPML,JTAG,Ref Clks… XAUI (PCIe?) 1 GbE IPML, JTAG, I2C Sensors I2C GbE (Debugging) TTC µ POD Tx (x4) µ POD Rx (x4) 48 Tx links (ADC Data@40MHz) (e/jFEX Data@40MHz) LTDB µ POD Tx (x4) µ POD Rx (x4) µ POD Tx (x4) µ POD Rx (x4) µ POD Tx (x4) µ POD Rx (x4) FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 FPGA (DSP SOC) 48 4 Up to 4 GBT connectors 1 1 1 FELIX GBT GbE ATCA Shelf Power 48V Power (Data Monitoring) ATCA Switches (ATLAS Event-TDAQ) (TTC) 10/100MbE 48 Rx links 48 Tx links 48 Rx links 48 Tx links 48 Rx links 48 Tx links 48 Rx links L1Calo (ADC Data@40MHz) LTDB L1Calo (ADC Data@40MHz) (e/jFEX Data@40MHz) LTDB L1Calo (ADC Data@40MHz) (e/jFEX Data@40MHz) LTDB L1Calo GBT 1GbE (ATCA Management, Monitoring) MMC 4 GBT 10GbE 1 1 1 XAUI 10Gbps (PCIe?) 1 4 GBT 1 GbE JTAG I2C IPML,JTAG,Ref Clks… Ref Clks (TTC, XAUI…) 1 1 JTAG

18. 124 LTDB’s total – 320 SuperCell channels per LTDB – 8 channels per fiber – 40 fibers per LTDB – 1 LTDB per AMC 124 AMC cards total – 4 AMC per Carrier Card – 31 Carrier Cards spread over 3 ATCA shelves LAr Accounting 18

19. ADDC to AMC – AMC pigtails – Connection to patch panel ? LDPB to Muon Sector Logic – Connector choice Base Interface: GbE (2 ports) – What: Reprogramming, Control, Monitoring, Histos? – How: switch between base interface and all components (AMC’s FPGA, Carrier FPGA)? Fabric Interface: 10GbE/40GbE? (2 ports) – What: Data monitoring by PC farm. – Where is it connected to: Carrier FPGA, switch and AMCs? – If connected to Carrier FPGA, what is between AMCs and Carrier FPGA? Interfaces (1) 19

20. GBT links – What: TTC, Busy, Data from AMC to TDAQ (FELIX) – Where to: AMC, Carrier FPGA – How many? – TTC clock extraction – TTC command transmission & Decoding. IPM Bus – Only ATCA functions? Power – Amount per Carrier, per AMC Interfaces (2) 20

21. IPML Bus /ATCA signals – Anything special? JTAG Bus – Keeping the chain when no AMC present? TTC/Busy ? – Clock – Commands – Busy GbE Fast links – 10 GbE, PCIe… GBT Power AMC/Carrier Interface 21

22. AMC 22 1GbE IPM-L JTAG TTC XAUI (PCIe?) 4 FPGA DDR3 FLASH 12 4x12 Optics Tx 4x12 Optics Rx DC/DC;LDO Oscillators MMC Sensors SDRAM Optical fibresAMC connector 1 Power : 12V 3.3V-IPMI PLL L1Calo e/jFEX Data @40MHz GBT 1 LTDB ADC Data @40MHz Ref Clks (TTC, XAUI…) Reset 4

23. LDPS TDR 23 LTDBs ADC Data@40MHz LDPBs L1Calo e/jFEX Data@40MHz GBT Links LTDB Configuration & Monitoring, TTC TTC Partition PM PC ATCA System Manager Data Monitoring TTC To TDAQ PC Farm DCS FE TTC DCS GBT Links Busy LTDB Configuration & Monitoring BE ATLAS Event-TDAQ, TTC 10/40GbE Network ATLAS Event-TDAQ LDPB Reprogramming & Configuration & Monitoring TDAQ Network GbE Network Shelf Managers ATCA Management & Monitoring DCS FELIX Custom Links