1. New ATCA compute node Design for PXD Zhen-An Liu TrigLab, IHEP Beijing Feb. 8 2011 ， 6th International Workshop on DEPFET Detectors and Applications

2. 2011-2-82 Outline Requirements to CN Approach 1: direct upgrade Approach 2: AMC-based Upgrade Example firmware development Summary 6th DEPFET Workshop in Bonn 2011

3. Requirements to CN PXD requirements summarized last workshop in Valencia – Based on Function of PANDA CN – 5 FPGAs/CN – 8 optical links/CN(3Gbps) – Ethernet links/CN – 20G DDR2 memory/CN (2G X 5) – Flash Memory – Improvements 6Gbps bandwidth per link 4GB DDR2 Memory per FPGA Approaches 1.Modification from present version 2GB DDR2-4GB DDR2, – Space not possible 3.125->6.5Gbps – Possible with SFP+ done 2.New design AMC/Carrier, xTCA compliant AMC card PCB received ATCA carrier under design 2011-2-836th DEPFET Workshop in Bonn 2011

4. 2011-2-8 4 Approach 1: modification from PANDA CN MODIFICATIONS – I2C communication capacitor – Voltage/Current monitor circuit – 3.125/6.5Gbps compliant Production done – 2 boards assembled with virtex4 fx60-11 for SFP+ PXD readout – 2 boards assembled with virtex4 fx60-10 for SFP 6th DEPFET Workshop in Bonn 2011

5. Test of New Boards Data flow – Optical link->DDR2 SDRAM -> PC Optical link@6.25Gbpslink@6.25Gbps – Tested about 1 hour, estimated BER< 1.0x10^-14 P2P transmission on backplane – Failed to transmit data@6.25Gbps, limited by bandwidth of backplane 2011-2-856th DEPFET Workshop in Bonn 2011 Eye-diagram of optical link Eye-diagram of backplane transmission

6. Evaluation Hardware 4 CN boards are ready for PXD testbenchs – 2 boards with 6.25G optical links – 2 boards with 3.125G optical links – All boards with 2GX5=10G DDR2 memories PXD evaluation can be done based on these boards – 4G DDR2 is not so necessary from Itoh-san – Further development (approach 2) will go No problem foreseen for April conclusion 2011-2-866th DEPFET Workshop in Bonn 2011

7. 2011-2-87 Approach 2: AMC-based Upgrade Compute Node – An universal high performance platform prepared for multiple applications – High availability: ATCA-based (full mesh topology in backplane)‏ – FPGA (now Virtex-4)‏, 5 Virtex4 FX60 FPGA, 10 GB DDR2 SDRAM (2GB per FPGA), High Bandwidth, 320MB FLASH (64MB per FPGA) – It is a prototype of HADES/PANDA But, not satisfied to BESIII, Belle II PXD and may be others systems in – Memory capacity – Input and inter Link rate – Timing and control capabilities – XTCA for physics compliant (IHEP is a founder of this standard) – … It is not good to design one board for all applications – Complexity – High cost (prototype) XTCA is a good choice for next generation – 1 ATCA Carrier Board with high bandwidth switch + AMC modules – Custom AMC modules for different applications – flexibility to reuse and upgrade – Flexibility for maintenance 6th DEPFET Workshop in Bonn 2011

8. 2011-2-88 Development of Carrier Board The mother board – based on xTCA – allowing backplane data transmission – 4 AMC connectors – FPGA0 for interconnections – IPMC routing – Clock/trigger/ distributions – Power conversions – RTM reservation Status – PCB layout ongoing 6th DEPFET Workshop in Bonn 2011

9. 2011-2-8 6th DEPFET Workshop in Bonn 2011 The structure of the xFP card(AMC) 2 x 3Gbps /FPGA 2 x 2G DDR2 1 Gbps ethernet 1 UART 1 IPMC/MMC 9

10. 2011-2-810 Module Sizes and Width six types of Module available – A Full-size Module is the most common, allowing up to 23.25 mm high components – A Mid-size Module allows component heights maxed at 11.65 to 14.01 mm – A Compact Module allows only 8.18 mm. – A special carrier card known as hybrid or cutaway carrier is required to hold one Full-size Module or two Compact-size – Each height is paired with a width, single (74mmx180mm) or double (149mmx180mm), describing how many carrier slots the board fills. A double width card allows more component space, but does not provide any additional power or bandwidth because it only uses a single connector. From PICMG AMC.0 R2.0

11. 2011-2-811 Hardware Platform Management Management is performed through IPMI messaging over an onboard IPMB referenced throughout this specification as IPMB-L. Each Module has a unique IPMB-L address derived from its Geographic Address. Carrier provides ways to isolate the IPMB-L connection to each Module. MMC: Module Management Controller From PICMG AMC.0 R2.0 6th DEPFET Workshop in Bonn 2011

12. PCB Layout of the xFP(AMC) PCB in hand Soldering to be done soon with XC5VLX50T Test to be done in Feb. 2011-2-86th DEPFET Workshop in Bonn 201112

13. PCB Layout of the xFP(AMC)-cont. PCB both sides 2011-2-86th DEPFET Workshop in Bonn 201113

14. PPC version of AMC XC5VFX70T with PPC – Order soon after back to work PCB compatible No problem in 2 months(Soeren’s concern) 2011-2-86th DEPFET Workshop in Bonn 201114

15. Firmware development Communication with DHH – Format not decided yet – Further discussion with Igor – Possible based on BES3/Belle2link? – Practice successful 2011-2-8156th DEPFET Workshop in Bonn 2011

16. Example Firmware Development Open source Linux Linux boot and reconfiguration online Data flow 2011-2-8166th DEPFET Workshop in Bonn 2011

17. Demonstration System 2011-2-8176th DEPFET Workshop in Bonn 2011 NFS Event Gen..cxx Data token.C macro Detector data.hex Reconstruction.cxx Data analysis.C macro framework Software COPPER PMC CPU FINESSE ComputeNode optical link@2Gbps

18. Plan for April Evaluation New CN with 6Gbps links is ready for use New design w AMC card will be ready end Feb. Carrier board ready in April MTCA shelf expected April SHOULD BE NO PROBLEM ! 2011-2-8186th DEPFET Workshop in Bonn 2011

19. 2011-2-819 Summary Test results show that new ATCA CN boards work well, Optical link @6.25Gbps is stable Example firmware for data transmission and readout developed AMC-based upgrade is ongoing smoothly Should be no problem for April evaluation! 6th DEPFET Workshop in Bonn 2011

20. Thanks for you attention !

21. backups 2011-2-8216th DEPFET Workshop in Bonn 2011

22. 2011-2-822 Power Distribution Payload power (AMC connector) – +12V(10.8V to 13.2V) – Max. 80W, 7.4A(@10.8V) Management power – +3.3V +/- 0.3V – Max. 150mA, At least 165mA(Carrier) BUT ATCA power supply: 200W/slot x80% = 160W – Only ~35W for each module if there are 4 AMC modules – ~55W for each module by increasing power to 300W/slot ( PIM300 + power module) – Cooling challenges for 300W+ power supply From PICMG AMC.0 R2.0 6th DEPFET Workshop in Bonn 2011