1. ROD Activities at Dresden Andreas Glatte, Andreas Meyer, Andy Kielburg-Jeka, Arno Straessner LAr Electronics Upgrade Meeting – LAr Week September 2009 1

2. Outline ROD trigger L1 and L2/EF buffers Concept for ROD and ROB/ROS Test Setup in Dresden Outlook 2

3. Expected data rates and buffer sizes for one ROD: at input: ~100 Gbps/FEB x 14 reduction by L1 trigger sums: ~ factor 10 (?) → ~ 140 Gbps to L1Calo (?) L1 buffer size (latency 5 μs): ~ 7.2 Mbit (?) assumption: 5-sample energy+time is calculated each 25 ns per cell: 16 bit energy/gain/parity + 20%*16 bit time/quality ~ 20 bit → *128 channels *14 FEB *40 MHz * 5 μs = 7.2 Mbit) L1 accept rate: ~200 kHz L2/EF output rate = ROB input rate: 8.6 Gbps → over ATCA backplane derived from the following expectation: 14 FEB * 128 channel * 16 bit * 1.2 time/quality * 1.25 (10/8 enc.) * 200 kHz = 8.6 Gbps maximum rate could however be: (based on today’s ROB max. input rate) (1.28 Gbps / 2 FEB) * 14 FEB * 200kHz/100kHz = 18 Gpbs L2 latency: ~ 40 ms ROB buffer size: 4.1 Gbit (for 12 RODs = 1 crate) 8.6 Gbps * 12 * 40 ms ROB output rate (data requested by L2/EF/DAQ): ~ 4 Gbps 0.3 Gbps / (12 links * 2 FEB) * 14 FEB * 12 ROD * 2 (occupancy) ROD Trigger Buffers and Data Rates 3

4. Communication between ROD and ROB/ROS 4 different options

5. ROB/ROS 5

6. Test Setup 6

7. ATCA 10GbE Test Setup 7 optical wiring (orange) and switch management (green, gray) ROD Switch Server test environment in Dresden: RadiSys Promentum ATCA Sys-6010 equipped with ROD Board and 10GE Switch incl. 10GE XFP transceiver Dual Xeon Server equipped with one 10GE dual- port Myricom NIC incl. optical SFP+ transceiver current achievements: external loopback between both ports of NIC (“Send-to-Self” Linux kernel patch needed) bandwidth w/ and w/o ATCA Switch: ~9.90 GBit/s with MTU 9000 (TCP/UDP Stream) next steps: implement and test Remote DMA: DMA access via Ethernet minimize CPU usage during memory access Server → NIC → (switch) → NIC → Server test performance limits of RDMA/Server-CPU: data transfer with 4 Myricom cards planned

8. 10 GbE Data Transfer With FPGA BNL ROD ATCA Blade FPGA (Xilinx Virtex5 - XC5VFX70T) 10-Gigabit Ethernet MAC Core ATCA -Back plane GTX- Trans ceive r 10-Gbit Eth. over XGMII Switch Blade 10 Gigabit Ethernet Switch Host PC User Logi c 10Gbit Eth. over XAUI IP-Core 10Gbit Eth. over XAUI 10-Gbit Eth. over Fiber 8 try 10 GbE data transfer with FPGA → possible format to send data to ROB/L2/DAQ interface possible architecture in ATCA crate:

9. 10 GbE Data Transfer With FPGA 9 R&D goal: use standardized protocol to be compatible with commercial components use commercial ROB based on ATCA blade with at least eight 10GbE links to fabric fast CPU 2-4 10GbE ports as interface to L2/EF/DAQ commercial 10 GbE MAC IP core from Xilinx very expensive free XAUI IP core exists from Xilinx want to try 10GbE MAC from opencores.org currently working on 1GbE some last problems to be solved many people are active to use 10GbE technology expect to get increased performance in future (next couple of years): 40-100 GbE does not exist now, but certainly in future today: CPU blades with two 10 GbE on fabric

10. Control Interface For ATCA Crate ATCA Crate PVSS Linux Host C based Appl. HPI client library HPI 100 MBit Ethernet file system before we received server PC and ROD prototype: learn about ATCA switch setup and crate control information PVSS used for control interface → reading of status info via Radisys HPI library

11. Simulation of Electronics and Signal Processing currently also thinking about digital fiter, L1 buffer and L1Calo interface: started to simulate electronic signals with pile-up goal: optimize readout timing / digitial filter / trigger tower size this is needed to get a new/optimal and possibly flexible mapping of cells to trigger sums will be important to estimate ROD-to-ROD and ROD-to-L1Calo rates and architecture first step: simple ROOT software for fast tests will be extended with more realistic pulse shapes and pile-up spectra single pulse and pile-up @ 10 35 cm -2 s -1 “ideal” pile-up and 20 GeV deposits digitally filtered signal

12. Summary and Outlook 12 we are learning to use ATCA and 10GbE performance with server PC and ATCA switch reaches 9.9 Gbps as expected next big projects: Remote DMA 10 GbE in FPGA continue electronics simulation → ROD/L1Calo interface