The Evaluation Tool for the LHCb Event Builder Network Upgrade Guoming Liu, Niko Neufeld CERN, Switzerland 18 th Real-Time Conference June 13, 2012

Outlines  Introduction to the LHCb Upgrade  Potential solutions for the DAQ upgrade and the event builder network (or called DAQ network)  The evaluation tool for the DAQ network and some tests done in the lab with this tool  Summary 2

LHCb Upgrade  Overall schedule: installation in the second long shut- down of the LHC in 2018, be ready for data taking in 2019  Design luminosity: cm -2 s -1  Trigger: LHCb opts for a fully flexible software trigger.  Hardware trigger will be removed  The whole detector will be readout at the LHC collision rate of 40MHz.  Low Level Trigger (LLT) : reuse current hardware trigger  tune the input rate to the computing farm from 1 – 40 MHz when the system is not fully ready for 40 MHz  cope with a staged DAQ system 3

LHCb Upgrade: Trigger 20 kHz Software Trigger Low Level Trigger (LLT) Input Rate: MHz to tape 40 MHz Event size: ~100 KB 4

Current LHCb DAQ  Readout board (ROB): custom FPGA board  UDP-like transport protocol MEP (Multi- Event Packet): several events packed into one message  Push scheme: ROBs assemble data fragments and send out the packets immediately  Deep buffer is required in the routers and the switches 5 Evt m Frag. Evt m Frag. Evt m Frag. CPU n: DataReq Evt m, Dest n Evt m, Dest n Evt m, Dest n

LHCb DAQ upgrade  Requirements  Event size: ~100 KB, readout rate: 40 MHz uni-directional throughput of the DAQ network: at least 32 Tbit/s  Two high-speed interconnection technologies  Ethernet (10G/40G/100G)  Infiniband (FDR/...)  Two topologies  Large core-router (current solution in LHCb) pros: “simple” architecture, good performance cons: expensive, not many choices  Fat-tree topology with many small Top-of-Rack (TOR) switches pros: cost-efficiency, scalability, flexibility cons: complexity 6

 Two DAQ schemes: push or pull  Push scheme with no traffic shaping  Push scheme with barrel shifter traffic shaping  Pull scheme: the destination sends data requests to the sources one by one 7 DAQ Schemes Diagram of barrel shifter traffic shaping

 Motivation  Demonstrate and verify the DAQ protocol  Measure the network performance and verify the solution based on different technologies and architecture 8 Evaluation tool for the network upgrade Network Transport Layer Data Flow manager Network Transport Layer Data Source Generator Event Builder Performance measurement Core  Basic components:  Core unit: dispatch jobs, handle exceptions,...  Data flow manager: control the data flow  Network transport layer: an abstract layer of the underlying network  Performance measurement: collect information from different modules

 Network: 10G Ethernet and QDR Infiniband  Server: dual-port 10G Ethernet card and 4x QDR Infiniband card 9 Setup in the lab Nodes 8 Type DELL R710 CPU Xeon E5520 at 2.27 GHz Memory 3 GB Network EthernetInfiniband Adapter Chelsio T420-CR dual-port 10GBASE-SFP, PCIe 2 x8 Qlogic HCA, qle7340 4x QDR PCIe 2 x8 Switch Voltaire Vantage 6048, 48 ports, 10 GbE Qlogic BS01, 36 ports, 4x QDR

 Tests in the lab so far  Push scheme  Push with barrel-shifter traffic shaping  Pull scheme  Infiniband network throughput measurement  So far, preliminary results only, the tool is to be optimized 10 Some Preliminary Tests

Study the current LHCb DAQ scheme in 10G Ethernet  Data flow manager send broadcast message to all servers using one 10G interface  Emulate the trigger packet  Synchronize all sources  All servers are data generators and receivers, using one interface to receive broadcast message, the other one to send and receive traffic. 11 Case 1: Push Trigger

 Main concern: buffer size in the network device.  Packet drop rate at different size of burst. Measure the packet drop rate at less than 50% of the line rate  Simple push  7 x 1 push: 7 sources to 1 destination  7 x 7 push: 7 sources to 7 destinations  7 x 7 push with barrel-shifter traffic shaping 12 Push: packet drop rate test

13 Push: results Message size: 13 kByte Packet drop rate: 4.3e-7 Message size: 14 kByte Packet drop rate: 1.4e-7 Message size: 50 kByte Packet drop rate: 1.3e-7

14 Case 2: pull  7 x 7 pull: 7 sources to 7 destinations using tcp/ip Performance degrades from message size 20 KB

 Network throughput for 1 to 1, 4 to 1, and 4 to 4 using RDMA (Remote Direct Memory Access) for data transfer 15 Case 3: Infiniband

 More studies for the LHCb event builder network upgrade are needed, so a generic evaluation tool will be very useful  A few basic modules has been used for the preliminary studies and the results have been shown  Push DAQ scheme on 10 GbE  Pull scheme on 10 GbE  Infiniband network throughput using RDMA 16 Summary

17