1. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU RP6 project JRA1 FutureDAQ RII3-CT-2004-506078 Data Acquisition Backbone Core J.Adamczewski, H.G.Essel, N.Kurz, S.Linev GSI, Experiment Electronics: Data Processing group Again: motivation What we have achieved since January (Legnaro) Performance measurements To do

2. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 2 Driving forces for DABC 2004 → EU RP6 project JRA1 FutureDAQ* 2004 → CBM FutureDAQ for FAIR 2005 → FOPI DAQ upgrade (skipped) * RII3-CT-2004-506078 1996 → MBS future 50 installations at GSI, 50 external http://daq.gsi.de Detector tests FE equipment tests Data transport Time distribution Switched event building Software evaluation MBS event builder General purpose DAQ Data Acquisition Backbone Core Intermediate demonstrator Requirements build events over fast networks handle triggered or self-trigger front-ends process time stamped data streams provide data flow control (to front-ends) connect (nearly) any front-ends provide interfaces to plug in application codes connect MBS readout or collector nodes be controllable by several controls frameworks

3. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 3 CBM data acquisition data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher switching network BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution W.F.J.Müller, 2004

4. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 4 1. A conventional LVL1 trigger would imply full displaced vertex reconstruction within fixed latency. 2. Strongly varying complex event filter decisions needed on almost full event data No common trigger! Self triggered channels with time stamps! Event filters CBM data acquisition New paradigm: switch full data stream into event selector farms 10 MHz interaction rate expected Time stamps in all data channels, typical resolution ~ns) required 1 TByte/s primary data rate (Panda < 100 GByte/s) expected 1 GByte/s maximum archive rate (Panda < 100 MByte/s) required Event definition (time correlation: multiplicity over time histograms) required Event filter to 20 KHz (1 GByte/s archive with compression) required On-line track & (displaced) vertex reconstruction required Data flow driven, no problem with latency expected Less complex communication, but high data rate to sort

5. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 5 CBM DAQ features summary Self-triggered time stamped data channels. Level 2 trigger complex: transport until filter farm FPGA controlled data flows Event building on full data rate ~1TB/s Event builder network BNet: ~1000 nodes, high-speed interconnections Linux may run on most DAQ nodes (even FPGAs)

6. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 6 PCIe FE Front end board: sampling ADCs, clock distribution Active Buffer board *: PCI express card PC ABB PCIe 2.5 GBit/s bi-directional (optical) link: data, clock Use case example: Frontend components test The goal: Detector tests FEE tests Data flow tests FE: Frontend board ABB: Active Buffer board * A.Kugel, G.Marcus, W.Gao, Mannheim University Informatics V

7. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 7 GE switch PC ABB PCIe DCB FE IB switch FE DC Front end board: sampling ADCs (4*4), clock distribution Data collector boards, clock distribution to FE Active Buffer board: PCI express card 8-20 PCs dual quad PC ABB PCIe *8 *4 Scales up to 10k channels, 160 CPUs bidirectional event building 2.5 GBit/s bi-directional (optical) link: data, clock 2.5 GBit/s data links Gigabit Ethernet The goal: Investigate critical technology Detector tests Replace existing DAQ FE: Frontend board DC: Data combiner board ABB: Active Buffer board GE: Gigabit Ethernet IB: InfiniBand MBS: MultiBranchSystem MBS Use case example: Frontend components test

8. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 8 frame buffer sorting tagging filter analysis frame buffer sorting tagging filter analysis det adc FPGA IB FPGA FEDCABB PC tasks GE analysis archive time archive PC FE: Frontend board DC: Data combiner board ABB: Active Buffer board GE: Gigabit Ethernet IB: InfiniBand Data flow

9. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 9 MBS topologies Readout 1 Readout n Triggerbus TCP Eventbuilder 1 Eventbuilder m Readout Eventbuilder Timebus TCP Timesorter Eventbuilder Readout Triggerbus Eventbuilder Readout Eventbuilder Monitor (analysis) Archive, RFIO Memory mapped bus n x m multiple EBs Time stamped (20 ns) EB hierarchy EB data transport ControlNode TCP < 20 MByte/s

10. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 10 MBS example topology Trigger Readout ADC Trigger Readout ADC Trigger Readout ADC Trigger Readout ADC Network Eventbuilder Detector Crates Multi Branch System GSI Standard Triggerbus

11. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 11 MBS: Example FOPI

12. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 12 Buffer2 (full) Buffer1 Slots MBS Readout Sender Trigger Event interrupt Buffer header Item header Item data (subevent) Item data Poll/yield TCP Trigger type Local event counter Copy/send MBS data channel to DABC Subevent pipe DABC

13. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 13 MBS node control and monitor External control asynchronous DAQ status Dispatcher Task commands (text) messages (text) Task Messageserver Statusserver asynchronous on request MBS node Task command thread working thread message thread Commands handled by threads Execution maybe in working thread Message thread maybe not needed Memory DIM server publish

14. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 14 data input sorting tagging filter analysis data input sorting tagging filter analysis IB Linux GE analysis archive Linux GE: Gigabit Ethernet IB: InfiniBand frontend DataDispatcher frontend other frontend MBS readout scheduler DABC overview DABC

15. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 15 DABC: Software objects of data flow engine DABC Module port Data manager DABC Module port process Queue Device Transport Device TransportQueue Network Module manager locally A module processes data of one or several data streams. Data streams propagate through ports, which are connected by transports and devices

16. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 16 Standard DAQ framework for LHC CMS experiment * C++ libraries on Linux, modular packages (SourceForge) Distributed xDAQ applications Configuration: XML Data transport: I 2 O protocol (Intelligent IO) Communication: http, cgi; SOAP messages InfoSpace: Global parameter access (subscription) State machines (sync/async FSM) Message logger, error handler Monitoring tool Hardware access library (HAL) Job Control (task handler for node control) others: exceptions, threads, data (de)serializers... The CMS xDAQ framework * Orsini, Gutleber http://xdaqwiki.cern.chhttp://xdaqwiki.cern.ch

17. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 17 Developments at GSI 2006 Developments at GSI for DABC (FDAQ) DABC application base classes prototypes xDAQ peer transport for InfiniBand (uDAPL) uDAPL buffers managed within XDAQ memory pool avoids memcopy and new buffer allocation for each send package: l ookup if posted memory reference is known as send buffer user code can write directly into uDAPL send buffer multiple threads for sending, releasing, and receiving buffer PCI/PCIe drivers* integrated in Hardware Access Library HAL DIM server as interface to xDAQ InfoSpace to control applications Test GUI in Labview over DIM Test GUI in Java speaking SOAP or DIM with xDAQ executives * Collaboration with Guillermo Marcus, Mannheim

18. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 18 Node XDAQ Executive (process, addressspace) Web server (SOAP), DIM** server XDAQ Application (thread), infospace DABC Core (threads) XDAQ* Infrastructure * Standard DAQ framework for LHC CMS experiment Orsini, Gutleber http://xdaqwiki.cern.chhttp://xdaqwiki.cern.ch ** DIM: Distributed Information Management System, CERN

19. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 19 Transport & Queue DABC: Synchronization DABC Module A processInput processCommand ActionEvent manager Thread with 1-n modules Command Device thread Transport & Queue A device, i.e. socket device controls several ports (transports). Once a queue buffer is filled, the transport signals the ActionEvent manager, which in turn calls the processInput function of the associated module. Commands are synchronized with the data flow through the ActionEvent manager

20. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 20 plugin input DABC: Example with module chain plugin Readout plugin combiner receiver sender Network plugin builder plugin analysis Node 6 different modules

21. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 21 Class diagram as currently implemented Ethernet InfiniBand PCIboard Port 6 Modules

22. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 22 Implementation of a PCI board dabc::Transport dabc::Device dabc::PCIDevice dabc::PCITransport dabc::PCIWorkingThread dabc::AbbDevice Driver components

23. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 23 Controls Tools for controls Configuration via XML files State machines, Infospace, message/error loggers, monitoring Communication: Webserver, SOAP, DIM Connectivity through DIM to: LabView, EPICS, Java, any DIM client/server Java with NetBeans (Matisse GUI builder) maybe soon in Eclipse Front-end controls? Mix of cooperating control systems First LabView and Java GUIs operable

24. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 24 Controls & monitoring communication XDAQ Executive (process, addressspace) Web server (SOAP) DIM client: Java GUI LabView GUI EPICS GUI SOAP client: Java GUI Infospace* XDAQ Application Modules command queue DABC Core DIM server State machine Web browser GRIDCC * Infospace: remotely accessible parameters

25. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 25 LabView-DIM Control * Generic construction of fixed parameter table from DIM servers * Dietrich Beck, EE

26. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 26 Java-DIM Control Generic construction from commands and parameters offered by DABC DIM servers Applications create the commands and parameters and publish Rate meter, trending, statistics

27. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 27 InfiniBand test cluster at GSI 4 nodes: –Double Opteron 2.2 GHz, 2GB RAM –Mellanox MHES18-XT host adapter (PCIe) –2x Gigabit Ethernet host adapters –SuSE Linux 9.3, x64bit version Mellanox MTS2400 24X InfiniBand switch InfiniBand switch: TS2400 InfiniBand HCA: MHES18-XT Gigabit Ethernet switch

28. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 28 InfiniBand software Mellanox IB Gold 1.8.0 –IPoIB: IP over InfiniBand driver –uDAPL: User Direct Access Programming Layer –MPI: Message Passing Interface OpenFabric Enterprise Distribution (OFED) 1.1 –IB Verbs (similar to uDAPL, plus Multicast) MVAPICH2: MPI 2 implementation for IB IBAdmin OpenSM (subnet manager) (for Multicast) Very extensive testing of all packages. Scheduled all to all data transfer (basic pattern for switched event building): 500-800 MByte/s (2K-64K buffer) per node. Cooperation with Forschungszentrum Karlsruhe established to scale the tests to 32-64 InfiniBand nodes, with UNI Mainz with 110 nodes After extensive simulations of event building Bnet with SystemC: Testing the reality

29. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 29 Network test performance 4x4 bidirectional

30. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 30 Optimized transfer (23 nodes at FZK), all to all Synchronization never overloads receivers, but gains 25% only! We thank Frank Schmitz, Ivan Kondov and Project CampusGrid at the Institute for Scientific Computing, Forschungszentrum Karlsruhe for providing resources and support for the large-scale measurements.

31. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 31 Scaling of performance (FZK) Degration only with large buffers

32. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 32 plugin input DABC: Full event building example measurement 4*Readout Data generators combiner receiver sender Network builderanalysis 4 nodes, all to all 6 different modules Node

33. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 33 Dataflow engine and event building rate 4x4 nodes

34. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 34 DABC further tasks Achieved Infrastructure –Setup, configure –Communication lines –Multi-threading Data flow engine –InfiniBand –Sockets (Gigabit Ethernet) –first PCIexpress board –very high performance –back pressure Very first (0.1) Java GUI Todo Data formats Combiner/Time sort MBS event building Event building Final Programming Interface definitions Documentation

35. DABCDABC D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 28.2.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 35 Resources and time People of data processing group H.G.Essel J.Adamczewski N.Kurz S.Linev People of controls group maybe one FTE People from CBM hopefully CBM requires in 2007 a data taking system Start with small system, grow on demand Preliminary controls NUSTAR ? In discussion MBS a first test bed