1. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev1

2. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev2 Outline Requirements of DAQ for CBM The InfiniBand Cluster, uDAPL XDAQ tests on the IB-Cluster Summary and further evaluation

3. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev3 CBM data acquisition data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher 1000x1000 switching BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution

4. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev4 CBM DAQ features Triggerless data transport until filter farm Event building on full data rate ~1TB/s B-net: ~1000 nodes, high-speed interconnections Linux may run on all DAQ nodes (even FPGAs) Test cluster with InfiniBand: small „demonstrator“ set-up within next year

5. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev5 InfiniBand test cluster 4 nodes: –Double Opteron 2.2 GHz, 2GB RAM –Mellanox MHES18-XT host adapter –2x Gigabit Ethernet host adapters –SuSE Linux 9.3, x64bit version Mellanox MTS2400 24X InfiniBand switch PCI Express x8

6. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev6 Testcluster networks InfiniBand switch: MTS2400 InfiniBand HCA: MHES18-XT Gigabit Ethernet switch Ethernet cabling InfiniBand cabling

7. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev7 InfiniBand software Mellanox IB Gold 1.8.0 –IPoIB: IP over InfiniBand driver –uDAPL: User Direct Access Programming Layer –MPI: Message Passing Interface –others...

8. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev8 Direct Access Transport Developed by DAT collaborative - http://www.datcollaborative.org http://www.datcollaborative.org Objectives: –Transport and platform (OS) independence –Define user- (uDAPL) and kernel-level (kDAPL) APIs Features: –Zero-copy data transfer –RDMA - Remote Direct Memory Access

9. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev9 uDAPL test application C++ wrapper for C-based uDAPL library Same program runs on all nodes All-to-all connectivity Time synchronization between nodes ~2 µs One master (deliver commands) and many slaves Several kinds of traffic schedule can be generated and executed Statistics like transfer rate, lost packets, schedule accuracy can be measured

10. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev10 All-to-all communication test Each node able to send data to any other node Message- and RDMA-based transfer are supported Master generate schedule and distribute it to all slaves At predefined time schedule execution is started Several schedules were tested: –One-to-all schedule –All-to-one schedule –All-to-all round-robin –All-to-all fix target masternode02node01node03

11. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev11 Scheduled test results Slight difference between message- and RDMA-based transfer Reasonable transfer rates starting from 16 KB buffers

12. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev12 Chaotic (schedule-less) transfer tests Packets send without schedule Next packet send when previous is completed Performance depends on receiving queue size For small packets performance 2 times better than scheduled approach Low CPU usage

13. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev13 Data Acquisition framework requirements Configuration of multiple nodes (connections and algorithms, database?) Controls and monitoring Data transport (format, protocol, hardware drivers?) Message logging Error handling and failure recovery (ARMOR?) Modular architecture, extensability! (sub-detector tests) User interface?

14. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev14 Standard DAQ framework for LHC CMS experiment (Orsini, Gutleber) http://xdaqwiki.cern.chhttp://xdaqwiki.cern.ch C++ libraries on Linux, modular packages Each node: context, xdaq process with embedded xdaq applications Configuration: XML Data transport: I2O protocol (Intelligent IO) Communication: http, cgi; SOAP messages The CMS XDAQ framework

15. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev15 XDAQ features continued... http://xdaqwiki.cern.chhttp://xdaqwiki.cern.ch State machines (sync/async FSM) Message logger, error handler Monitoring tool Hardware access library (HAL) Front End Driver (FED kit, for CMS!) Job Control (task handler for node control) others: exceptions threads infospace data (de)serializers,... The CMS XDAQ framework

16. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev16 XDAQ on the CBM InfiniBand cluster Installation problems: XDAQ distribution not 64 bit save! Code adjustments required kernel modules for CERN linux excluded! drivers for CMS hardware (FED) excluded coretools done, worksuite partially,…  Basic evaluation possible!

17. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev17 XDAQ tests on CBM InfiniBand cluster Tested features on IB cluster: http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus http://wiki.gsi.de/cgi-bin/view/Daq4FAIR/DaqClusterSoftwareXdaqStatus hyperdaq webserver (main test controls ui) xml configuration setup control variable access and export (infospace) SOAP messaging (test controller application) state machines (sync./async. with web/SOAP ui) monitoring application (very raw ui!) multithreading (workloop framework) exceptions, data wrappers, message logging, toolbox,… i2o data transport (tcp roundtrip example)

18. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev18 XDAQ: web server hyperdaq web interface

19. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev19 XDAQ executive with applications Common XML configuration file: Each node knows all applications on all nodes! Unique addressing by context (url) and application id

20. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev20 XDAQ: SOAP messaging xrelay: send SOAP control messages example commands: Configure, Enable, Halt, Reset,.. any new commands may be defined web interface not suited as real control system UI! XDAQ applications may exchange SOAP messages with other UI (Labview?)

21. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev21 XDAQ: messaging and peer transport peer transport i2o tcp fifo http SOAP messages transport layer DAPL new development

22. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev22 XDAQ: I2O messaging and peer transport

23. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev23 Peer Transport DAPL (ptdapl) GSI development 03/2006-05/2006 C++ wrapper class for uDAPL C library (class TBasic) peer transport tcp as „template“ (starting point) uDAPL buffers managed within XDAQ memory pool (class TEndpointBuffer) avoids memcopy and new buffer allocation for each send package: l ookup if posted memory reference is known as send buffer user code can write directly into uDAPL send buffer multiple threads for sending, releasing, and receiving buffer

24. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev24 Peer Transport DAPL structure uDAPL peer transport senderreceiver transmitter messenger receiverloop buffer pool message (I2O) channel

25. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev25 I2O roundtrip on InfiniBand cluster Standard XDAQ roundtrip benchmark: sender node posts I2O frame to receiver node receiver callback sends frame back to sender sender callback sends frame back again, etc… perfmeter class records bandwidth, latency,… web display of results on sender node (java!) Code independent of transport implementation! XML configuration file does set up (pttcp eth, pttcp IB, ptDAPL, ptFIFO,...)

26. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev26 XDAQ executive MyRoundTrip HyperDAQ I2O roundtrip setup sender receiver XDAQ executive MyRoundTrip token() EnableAction() default() I2O callback web application state machine Peer Transport

27. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev27 I2O sender/receiver benchmark New applications MyDataSource, MyDataDrain: sender node posts I2O frames to receiver node in workloop thread receiver callback gets frame and checks it multiple senders/receivers possible (n x m) perfmeter class measures on all nodes web display of results on all nodes (java!) Code independent of transport implementation! XML configuration file does set up (pttcp eth, pttcp IB, ptDAPL, ptFIFO,...)

28. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev28 HyperDAQ I2O sender/receiver setup senderreceiver XDAQ executive MyDataSource Benchmark() EnableAction() default() workloop (thread) web application state machine XDAQ executive MyDataDrain token() I2O callback Peer Transport default()

29. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev29 Transport bandwidth B and latency  P: package size  : transfer time („latency“) Observation:  linear with P:  B P P P big: network limit P small: framework latency !

30. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev30 ptDAPL benchmark results TestNbufs,rcv  0 (µs) C (µs/kByte) 1/  0 (kHz) Bmax (MByte/s) MyRoundTrip205.531.069181935 MySourceDrain500.991.0371010964 MySourceDrain1001.71.017588983 MySourceDrain2001.831.013546987 MySourceDrain10001.221.029820972

31. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev31 Benchmark summary 955 MB/s plain uDAPL 234 MB/s XDAQ, I2O, TCP over IB 905 MB/s XDAQ, I2O, ptDAPL

32. IT/EE Palaver - 20.06.06FAIR DAQ - J.Adamczewski, S.Linev32 Summary and Outlook CBM DAQ requires fast builder network: InfiniBand test cluster IB hardware and software installed, tested,... uDAPL: developed C++ wrapper and test applications many tests with different traffic patterns were performed more tests on bigger InfiniBand clusters are required XDAQ as software framework: tested features: SOAP, web server, data transport,... I2O implementation for uDAPL done! cluster configuration? control system? scalability? hardware access?