1. Development of an ATCA IPMI Controller Mezzanine Board to be used in the ATCA developments for the ATLAS Liquid Argon upgrade Nicolas Dumont Dayot, LAPP on behalf of the ATLAS Liquid Argon Calorimeter Group

2. 27/09/20112ATCA IPMI Controller Mezzanine Board - TWEPP 2011 Outline ■ Context ■ Motivations ■ ROD evaluator ■ ATCA test board ■ ATCA controller mezzanine ■ Summary

3. Context 27/09/20113ATCA IPMI Controller Mezzanine Board - TWEPP 20113 Analog Pipeline (SCA) ADC 5MHz 16x12-bits Σ Calibration FRONT END ELECTRONIC 128 cells / Front End Board L1 Trigger 100kHz max BACK END ELECTRONIC Read Out Driver Read Out Subsystem 1 Optical Link @1.6 Gbps 800 Optical Links ADC 40MHz 128x18-bits Mux / Serializer Calibration FRONT END ELECTRONIC FEB L0/L1 Trigger 12 Optical Links @10Gbps ?? xx Optical Links x1600 x192 8 FEB/ROD Up to 14 FEB/ROD ?? 1 12 Future Today Read Out Driver Read Out Subsystem E,Eτ,χ² 100Gbps needed 128 cells / Front End Board FEB BACK END ELECTRONIC

4. 27/09/20114ATCA IPMI Controller Mezzanine Board - TWEPP 2011 We need : - High input and output bandwidth - High speed communications between boards - Powerful signal processing This is why ATCA platform was chosen: - High speed and high density communications between boards over backplane - Large boards (32x28 cm) - Possibility of rear transition modules (RTM), and mezzanines card (AMC) - Reliability (hot swap, real time diagnostic) We want to build an ATCA board to evaluate: - ATCA specifications (Intelligent Platform Management Interface ‘IPMI’ facilities) - Management trough Ethernet (Firmware upgrade, DSP configuration, monitoring…) - Many 12x10Gbps incoming optic fibers - High speed and high density board. - High power DSP cells from FPGA For that we have designed a board called “ROD evaluator” and an IPM Controller Mezzanine board Motivations

5. FPGA building block 27/09/20115ATCA IPMI Controller Mezzanine Board - TWEPP 2011 IPMC & board management ROD evaluator Synoptic

6. 27/09/20116ATCA IPMI Controller Mezzanine Board - TWEPP 2011 CAD completed : - 16 layers - Minimum lines width : 75 µm - Laser and blind buried vias Before launching production : -> Test FPGA building blocks -> Test FMC mezzanine with IPMC We have designed 2 boards: -> ATCA controller mezzanine board -> ATCA test mother board Cost =$$$$$ ROD evaluator Design

7. 27/09/20117ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Test board Insertion Switch Power Supplies Zone 1 Zone 2 Ethernet Sensors FPGA ARRIA IIGx CPLD Flash DDR3 POL Supplies ATCA Controller Mezzanine (IPMC & Config) IPMBus Ethernet ATCA blue led ATCA led 1 & 2 Update Channel Base Interface Tests of : FPGA building block - Boot from CPLD & parallel Flash - Communications with DDR3 & Flash ATCA CTRL Mezzanine : - IPMC (IPM Control) trough IPM Bus => Communication with Shelf manager - ATCA power supplies management => Hot swap (insertion switch) => Enable DC/DC - Alarm/failure diagnostic - Board configuration through Ethernet => Firmware upload => Optimal filtering coefficient upload => Sensor reading => Etc.. FPGA building block EEPROM Temp. ATCA test board Synoptic

8. 27/09/20118ATCA IPMI Controller Mezzanine Board - TWEPP 2011 Tests done : Power supply -> Sequence to fix FPGA silicon bug FPGA boot from CPLD and FLASH -> Selection of 2 firmwares in FLASH FLASH access with the FPGA -> NIOS DDR3 communication -> Access up to 1000Mb/s LVDS links with FMC through connectors -> Data rate up to 400Mb/s Next steps : - JTAG chain (multiplexer) - Ethernet link on Base interface (J2) - Sensor readings (SPI) - FPGA FMC protocol CPLD Flash DDR3 ArriaIIGx ATCA Controller Mezzanine Emerson ATC250 DC-DC converter J2 Update Channel J2 Fabric, Base interface J1 Power, IPMBus ATCA test board Mother board tests

9. 27/09/20119ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine Hardware FMC (FPGA Mezzanine Card) : - High pin count - Up to 160 links (74 differential links) FPGA : - Xilinx Spartan 6 : highly configurable I/O - Boot from SPI Flash - All FMC connector I/O driven by FPGA -> Bridge to the external world -> single lines or LVDS signals µC : - ARM cortex M3 processor : TI LM3S9B92 - Ethernet/USB/JTAG interfaces - IPMC implementation - JTAG master implementation 69mm 76.5mm Ethernet FPGA Spartan 6 µC LM3S9B92 bus FMC Mezzanine Power Supplies Connector SPI FLASH X 2 IO USB JTAG I2C IPMBus A&B µC FPGA IO

10. 27/09/201110ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine Software IP stack library IPM Controller JTAG Controller Board configuration Monitoring Hardware library IPMC module µC LM3S9B92 (Texas Instruments) & FPGA (Xilinx) Debugger GNU GDB & OpenOCD Compiler Gnu GCC : ARM Development tools : LINUX Web interface User environment TCP/IP client interface File server (boot and board config.) JTAG/USB IPMBus to Shelf manager Ethernet JTAG module Programmer T.I

11. 27/09/201111ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine Hardware tests Ethernet FPGA Spartan 6 µC LM3S9B92 bus FMC Mezzanine Power Supplies Connector SPI FLASH X 2 IO USB JTAG I2C IPMBus A&B FPGA interfaces: Tests done : - Boots from configuration SPI Flash - SPI Flash loaded by µC - µC FPGA communications - LVDS signals data rate up to 400Mb/s - External I2C temperature sensor reading Next steps: - External I2C EEPROM management - DC/DC converter I2C management - Protocol with FPGA on carrier board - 2 nd SPI Flash management µC interfaces: Tests done : - USB/JTAG/Ethernet : boot from Ethernet Tests on going: - IPMC (described in next slide) - JTAG master (described in next slide) Next steps : - Configuration and monitoring of the carrier board

12. 27/09/201112ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine JTAG master JTAG control: - JTAG chain controlled by either external connector or µC configured as JTAG master - Programming through Ethernet (via µC) µC as JTAG master : tests done -.xsvf files player validated to program the FPGA -.jam files player to program parallel FLASH via CPLD abandoned => not enough memory space in µC => programming time too long => the Flash will be uploaded through the FPGA and not the CPLD => the CPLD only help to boot the FPGA at power up µC as JTAG master : tests on going - JTAG chain multiplexer configuration on mother board => SCANSTA111 configured by JTAG commands

13. 27/09/201113ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine IPM Controller Specifications : - PICMG 3.0 R3.0 – ATCA base specification - IPMI v1.5 and relevant subset of IPMI v2.0 Development : - Based on open source CoreIPM - IPMB-L (IPM Bus local for AMC) not supported : suited only for ATCA board ATCA state machine : almost validated - Hot swap (handle switch), hardware address, ATCA LED, DC/DC enable, temperature sensor - Mother board EEPROM with board information not yet handled - Under stress test to detect reliability issues and improve robustness => state machine code shall never fail => tested with power up/down cycles every 5s during 24 h Event generator : next step - Events sent to Shelf manager in case of alarm/failure

14. 27/09/201114ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine FRU & SDR generator FRU (Field Replaceable Unit) data : - Identity of the carrier board =>Ex : IO description on J2 connector SDR (Sensor Data record) : - Sensors description of the carrier board =>Ex : handle switch, temperature sensor FRU generator : done - Will be used to load the EEPROM SDR generator : next step Definition area M4 preprocessor Definition user C structure File to EEPROM FRU GENERATOR

15. 27/09/201115ATCA IPMI Controller Mezzanine Board - TWEPP 2011 ATCA Controller mezzanine Example of info to Shelf manager FRU : current and previous ATCA state, ID, version,…

16. 27/09/201116ATCA IPMI Controller Mezzanine Board - TWEPP 2011 Summary We have designed an ATCA ROD Evaluator board to evaluate the functionalities needed by the next generation ROD in the context of the ATLAS LAR upgrade: - Large number of fast links (High speed and high density board) - Recent generation of FPGA (many DSP cells and Ser-Des) and optical receivers - ATCA platform For the control of that board, we have designed a generic ATCA IPMI Controller mezzanine in the FMC format. - Acts as an IPM Controller - Direct Ethernet access - Uploads the FPGA firmware and other parameters - Monitors the board parameters This mezzanine and its software is currently tested on an ATCA Test board we have designed Tests performed have been successful, but much work still need to be done !!!