New ATCA compute node Design for PXD Zhen-An Liu TrigLab, IHEP Beijing Feb ， 6th International Workshop on DEPFET Detectors and Applications

Outline Requirements to CN Approach 1: direct upgrade Approach 2: AMC-based Upgrade Example firmware development Summary 6th DEPFET Workshop in Bonn 2011

Requirements to CN PXD requirements summarized last workshop in Valencia – Based on Function of PANDA CN – 5 FPGAs/CN – 8 optical links/CN(3Gbps) – Ethernet links/CN – 20G DDR2 memory/CN (2G X 5) – Flash Memory – Improvements 6Gbps bandwidth per link 4GB DDR2 Memory per FPGA Approaches 1.Modification from present version 2GB DDR2-4GB DDR2, – Space not possible >6.5Gbps – Possible with SFP+ done 2.New design AMC/Carrier, xTCA compliant AMC card PCB received ATCA carrier under design th DEPFET Workshop in Bonn 2011

Approach 1: modification from PANDA CN MODIFICATIONS – I2C communication capacitor – Voltage/Current monitor circuit – 3.125/6.5Gbps compliant Production done – 2 boards assembled with virtex4 fx60-11 for SFP+ PXD readout – 2 boards assembled with virtex4 fx60-10 for SFP 6th DEPFET Workshop in Bonn 2011

Test of New Boards Data flow – Optical link->DDR2 SDRAM -> PC Optical – Tested about 1 hour, estimated BER< 1.0x10^-14 P2P transmission on backplane – Failed to transmit limited by bandwidth of backplane th DEPFET Workshop in Bonn 2011 Eye-diagram of optical link Eye-diagram of backplane transmission

Evaluation Hardware 4 CN boards are ready for PXD testbenchs – 2 boards with 6.25G optical links – 2 boards with 3.125G optical links – All boards with 2GX5=10G DDR2 memories PXD evaluation can be done based on these boards – 4G DDR2 is not so necessary from Itoh-san – Further development (approach 2) will go No problem foreseen for April conclusion th DEPFET Workshop in Bonn 2011

Approach 2: AMC-based Upgrade Compute Node – An universal high performance platform prepared for multiple applications – High availability: ATCA-based (full mesh topology in backplane)‏ – FPGA (now Virtex-4)‏, 5 Virtex4 FX60 FPGA, 10 GB DDR2 SDRAM (2GB per FPGA), High Bandwidth, 320MB FLASH (64MB per FPGA) – It is a prototype of HADES/PANDA But, not satisfied to BESIII, Belle II PXD and may be others systems in – Memory capacity – Input and inter Link rate – Timing and control capabilities – XTCA for physics compliant (IHEP is a founder of this standard) – … It is not good to design one board for all applications – Complexity – High cost (prototype) XTCA is a good choice for next generation – 1 ATCA Carrier Board with high bandwidth switch + AMC modules – Custom AMC modules for different applications – flexibility to reuse and upgrade – Flexibility for maintenance 6th DEPFET Workshop in Bonn 2011

Development of Carrier Board The mother board – based on xTCA – allowing backplane data transmission – 4 AMC connectors – FPGA0 for interconnections – IPMC routing – Clock/trigger/ distributions – Power conversions – RTM reservation Status – PCB layout ongoing 6th DEPFET Workshop in Bonn 2011

th DEPFET Workshop in Bonn 2011 The structure of the xFP card(AMC) 2 x 3Gbps /FPGA 2 x 2G DDR2 1 Gbps ethernet 1 UART 1 IPMC/MMC 9

Module Sizes and Width six types of Module available – A Full-size Module is the most common, allowing up to mm high components – A Mid-size Module allows component heights maxed at to mm – A Compact Module allows only 8.18 mm. – A special carrier card known as hybrid or cutaway carrier is required to hold one Full-size Module or two Compact-size – Each height is paired with a width, single (74mmx180mm) or double (149mmx180mm), describing how many carrier slots the board fills. A double width card allows more component space, but does not provide any additional power or bandwidth because it only uses a single connector. From PICMG AMC.0 R2.0

Hardware Platform Management Management is performed through IPMI messaging over an onboard IPMB referenced throughout this specification as IPMB-L. Each Module has a unique IPMB-L address derived from its Geographic Address. Carrier provides ways to isolate the IPMB-L connection to each Module. MMC: Module Management Controller From PICMG AMC.0 R2.0 6th DEPFET Workshop in Bonn 2011

PCB Layout of the xFP(AMC) PCB in hand Soldering to be done soon with XC5VLX50T Test to be done in Feb th DEPFET Workshop in Bonn

PCB Layout of the xFP(AMC)-cont. PCB both sides th DEPFET Workshop in Bonn

PPC version of AMC XC5VFX70T with PPC – Order soon after back to work PCB compatible No problem in 2 months(Soeren’s concern) th DEPFET Workshop in Bonn

Firmware development Communication with DHH – Format not decided yet – Further discussion with Igor – Possible based on BES3/Belle2link? – Practice successful th DEPFET Workshop in Bonn 2011

Example Firmware Development Open source Linux Linux boot and reconfiguration online Data flow th DEPFET Workshop in Bonn 2011

Demonstration System th DEPFET Workshop in Bonn 2011 NFS Event Gen..cxx Data token.C macro Detector data.hex Reconstruction.cxx Data analysis.C macro framework Software COPPER PMC CPU FINESSE ComputeNode optical

Plan for April Evaluation New CN with 6Gbps links is ready for use New design w AMC card will be ready end Feb. Carrier board ready in April MTCA shelf expected April SHOULD BE NO PROBLEM ! th DEPFET Workshop in Bonn 2011

Summary Test results show that new ATCA CN boards work well, Optical is stable Example firmware for data transmission and readout developed AMC-based upgrade is ongoing smoothly Should be no problem for April evaluation! 6th DEPFET Workshop in Bonn 2011

Thanks for you attention !

backups th DEPFET Workshop in Bonn 2011

Power Distribution Payload power (AMC connector) – +12V(10.8V to 13.2V) – Max. 80W, Management power – +3.3V +/- 0.3V – Max. 150mA, At least 165mA(Carrier) BUT ATCA power supply: 200W/slot x80% = 160W – Only ~35W for each module if there are 4 AMC modules – ~55W for each module by increasing power to 300W/slot ( PIM300 + power module) – Cooling challenges for 300W+ power supply From PICMG AMC.0 R2.0 6th DEPFET Workshop in Bonn 2011