E. Hazen -- ACES ACES 2011 MicroTCA in CMS E. Hazen, Boston University for the CMS collaboration

E. Hazen -- ACES MicroTCA in CMS ● What is MicroTCA? ● Common MicroTCA platform for CMS ● Hardware aspects ● Software aspects ● Summary and plans

E. Hazen -- ACES AdvancedTCA and MicroTCA AMC (double-width) AMC Advanced Mezzanine Card (AMC) Zone 1 Power Management Zone 2 Clocks Fabric up to 10 Gb/s serial links ATCA Module: 8U x 280mm AMC (uTCA): 73x180mm 149x180m m ATCA 8U Carrier AMC mezzanines uTCA Small crate AMC mezzanines (plug to backplane)

E. Hazen -- ACES ATCA and uTCA Features ● Card management functions using IPMB (I2C) bus ● Management power (always on) and payload power (switched) ● High-speed (up to 10Gb/s) backplane “fabric” ports (up to 21 per module) ● Flexible (i.e. unspecified) backplane topology ● Wide industry support (should evolve to low cost)

E. Hazen -- ACES CMS MicroTCA Common Platform in principle all CMS MicroTCA crates will use common... ● Hardware: ● 12 Slot redundant crate with dual-star backplane ● 12 AMC modules (double-width, full height) ● 1 Commercial MCH for Management / Ethernet ● 1 Custom MCH for Clock/Timing/DAQ ● Firmware / Software: ● IPMI use for configuration control / monitoring ● Ethernet protocol for control / local DAQ – Replacement for CMS “HAL” VME library – UDP / TCP yet to be decided

E. Hazen -- ACES Hardware Aspects

E. Hazen -- ACES CMS uTCA Readout Crate (i.e. HCAL) 12 AMC Slots Commercial MCH Management Ethernet Custom MCH Clocks Fast controls DAQ

E. Hazen -- ACES  TCA Dual-Star Backplane MCH 1 Commercial /Std MCH 2 aka “AMC13” Custom design for CMS Bi-directional serial (up to 10Gb/sec) point-to-point links from each AMC to MCH (redundant links to each MCH) CMS Use Fabric A (1 link) 2-4 Gb/s Fabric B (1 link) LVDS TTC Fabric D-G (4 links) Spare CLK1 MLVDS LHC clock Note: Interconnections can be customized by the backplane manufacturer inexpensively. Fabric A (1 link) Gigabit Ethernet Fabric B (1 link) Spare Fabric D-G Spare CLK1 Spare

E. Hazen -- ACES Example AMC -- HCAL uHTR ● Main features: ● Single Virtex 6 supports 36 optical and 24 backplane SERDES ● (DDR3) SDRAM for extended L1 pipeline storage (selective readout delay) XC6VLX240T or similar 12 link pairs System mcontroll er I2C System Control Bus SNAP12 12-way optical receiver SNAP12 12-way optical receiver Commercia l MCH Global DAQ LHC Clock Fast Controls DTC SNAP12 12-way optical transmitter SDRAM

E. Hazen -- ACES Prototype AMC (HCAL miniCTR2) MicroTCA Backplan e Connector Virtex 5 FPGA SFP Optical Transceiver s

E. Hazen -- ACES Spartan 6 Virtex 6 GT X SFP GT X SFP GT X SFP GT X IO SFP? CD S 128 Mbyte DDR3 GT P Flash IO TTC DAQ 6Gb/s Spar e MMC uC IPMI GbE JTAG LED s MCH1 Front Panel via T3 Fabric A 12 ports £ GB/s Fabric B 80 MB/s CLK F/O 40.xx CLK To AMCs CMS Custom MCH (Evolving) Optional RAM DAQ 6Gb/s 2:1 Switc h

E. Hazen -- ACES Custom MCH Board Stack T1 T2 T3 T4 New custom T1 base board MMC functions (as AMC-13) TTC optical rx 3 or 4x SFP Cross-over GbE from MCH1 for controls and local DAQ Clocks board Clock / controls fanout T3 board Provides JTAG / LEDs on front panel Crosspoint switch for trigger applications (could be omitted if some other solution is found for front panel) ● Base configuration has only tongues 1, 2 ● Base board - With optics and HS links (Fabric A) ● Clocks board - distributes LHC clock and controls ● Mezzanine connector for T3 with I2C ● T3 has JTAG and LEDs Connector to T3 provides: Power Serial bus (SPI or I2C) from MMC Front-panel signals: JTAG or SPI for miniUSB Status LEDs Quad SFP+ Cage

E. Hazen -- ACES Initial Custom MCH Prototype NAT-MCH Base Board (will not be used in final design) Fabric Board (DAQ) (move from T3 to T1 in final design) Clock / Controls Board (smaller in final design) Optics Connector Board (move to T1 in final design) Prototyped in 2010 ● TTC receiver tested ● Fiber optics tested ● Backplane links tested ● Integrated in system for HCAL test beam

E. Hazen -- ACES Software Aspects

E. Hazen -- ACES Evolution, not Revolution Current VME New uTCA

E. Hazen -- ACES Initialization Concerns ● Goal: ● uTCA crates with AMCs should power up into an operational status without “software intervention”. ● Two regimes 1) Commissioning, debugging: PVSS is not the environment for debugging. – Commissioning / configuration / debugging via IPMI GUI programs ( provided by crate / AMC manufacturer ) – Bulk (re) configuration : to be defined ● Most likely scripts 2) Operation: fully integrated into the experiment operations => SCADA interface ( PVSS for logging and transfer to offline data base) – GUI and API should be common for all uTCA crates

E. Hazen -- ACES One Rack PC controls a few uTCA crates On the rack PC several processes can run (tbd) : ● DAQ application ● openIPMI to DIM server ● FPGA configuration ● Access control (simplify the IP interface on the AMC) Processes and hierarchy TBD

E. Hazen -- ACES IPMI  DCS, DAQ ● Minimize the DCS (PVSS) complexity ● Monitor temperatures, voltages, currents ● Switch on/off crates ● Switch on/off AMC payload power ● Alerts (set Alerts levels) ● For test stands PVSS is not require (use IPMI GUI) ● Interface (i.e.) LHCb computer farm control uTCA  RCMP  (openIPMI)+DIM  (DIM+PVSS) ● Long term we want to remove DIM ● No direct communication between IPMI and DAQ

E. Hazen -- ACES Common FPGA configuration scheme ● For operations only one path for : port 0 (Ethernet) ● No direct configuration of the FPGA. Is not needed for operations. ● Use multi configuration images in flash, with boot image that is able to program the FLASH(es) ● Use hard coded boot logic of Xilinx and Altera. ● FLASH should be programmable by the MMC (IPMI compatible) ● In case of corrupted image 0. ● Common development of image 0 and FLASH configuration software

E. Hazen -- ACES Status and Plans ● Current Status ● Crate / interconnect spec document in draft ● Initial prototypes of MCH and AMC built (HCAL) ● Design of full prototype MCH underway ● Plans ● Install test crate for parasitic running in Dec 2011 ● Install “vertical slice” test system in HCAL in 2013 shutdown ● Support other subsystems upgrade efforts