Presentation is loading. Please wait.

Presentation is loading. Please wait.

First release of Data Acquisition Backbone Core

Similar presentations

Presentation on theme: "First release of Data Acquisition Backbone Core"— Presentation transcript:

1 First release of Data Acquisition Backbone Core
J. Adamczewski-Musch, H.G. Essel, N. Kurz, S. Linev GSI Darmstadt, Germany Experiment Electronics: Data processing group

2 FAIR accelerators FAIR baseline* SIS100 SIS300 p-linac HESR Super- FRS
p-bar target p-linac Super- FRS SIS100 SIS300 HESR CR RESR Unilac SIS 100 NESR Antiproton Prod. Target SIS18 Upgrade SIS 300 Accelerator SuperFRS PANDA Atomic Phys. Plasma Phys. Low Energy Exp. High Energy Exp. NESR Exp. FLAIR Atomic Physics HADES & CBM Experiment FAIR baseline* Facility for Antiproton and Ion Research *FAIR Monthly, April 2009

3 CBM experiment at FAIR Transition Radiation Electro- Detectors
Dipole magnet Ring Imaging Cherenkov Detector Transition Radiation Detectors Resistive Plate Chambers (TOF) Electro- magnetic Calorimeter Silicon Tracking System Projectile Spectator (Calorimeter) Micro Vertex One of experiments in FAIR will CBM – Condensed Barion Matter experiment Free-running electronic Precise timing for all detectors

4 DAQ architectures Conventional DAQ CBM DAQ HLT HLT FLES
L1 trigger HLT HLT FLES CBM DAQ Self-triggered Front-end all hits shipped to DAQ. Data push architecture Detector FEE buffer Readout buffer Switch Processor farm Storage Fast links Readout buffer outside radiation area. Many Gbyte storage easily possible. Allows L1 decision times of ms High-throughput Event building One cannot make filter (trigger) decision before complete event is reconstructed. Event reconstruction based on time stamps (time slices building) ~1 TB/s of primary data rates up to event builders First event selection done in processor farm.

5 InfiniBand performance studies
> 500 MBytes/s bidirectional + 25% InfiniBand is probable candidate for use in event building. Has low-latency and high bandwidth, implements zero-copy concept Scales with number of nodes, low CPU usage Meet CBM requirements thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz

6 Hardware tests nXYTER, 128-channel self-triggered
readout chip with few ns time resolution, DEFNI collaboration Beside big plans for final system now we need to perform a lot of detectors tests We think, that DAQ software components should be used also at that point for better integration into the final system. For instance, we start investigation of self-triggered electronics with nXYTER chip. The n-XYTER is a 128 channel asynchronous, self- triggered, self-sparcifying readout ASIC developed as a front-end for neutron scattering detector applications. In FAIR experiments like CBM it is used as prototype and test-bed of front-end electronics for several kinds of detectors prototypes. The CBM readout controller (ROC), developed in Kirchhoff Institut für Physik, Heidelberg, is an FPGA-based board, designed to read time-stamped data from n-XYTER and transfer that data via optical link to PC. As alternative option, Ethernet can be used for data transfer. CBM Readout controller (ROC), Kirchhoff Institut für Physik, Heidelberg

7 Future DAQ requirements
Flexible – adopt different kinds of soft- and hardware Compact – use only necessary code Scalable – from small detector tests to 100-nodes clusters Performing – low framework overhead Multiprocessing & multithreading I show only plans for CBM DAQ, but many other future experiments will demand for new features which are not provided by with our current DAQ system. Therefore we start development of new framework, which should fulfill such a new requirements. First problem which should be solved by DAQ is appropriate usage of multithreading capabilities of modern CPUs.

8 Multithreading Deadlock! Thread1 Thread2 runs runs Thread3 Thread4
void MainLoop() { while (IsWorking()) { writeSomething(); } runs void MainLoop() { while (IsWorking()) { doSomething(); } runs Deadlock! Thread3 Thread4 void MainLoop() { while (IsWorking()) { calcSomething(); } runs void MainLoop() { while (IsWorking()) { readSomething(); } runs Now it is unavoidable to use multi-CPU/multi-core machines Different tasks should run as different threads Threads is just place of code, but not code itself In simple case it is just kind of main loops, which running any code. As long as such threads are not communicate with each other, there is no problems. But if they should communicate, it often leads to different problems. For instance, dead locks. To avoid them, you should really count locks, check their hierarchy and so on. As a result, development of multithreading applications can become very complicated. To avoid such problem in DABC, another approach was taken.

9 Multithreading in DABC
Processor Events queues Events Timeouts Commands runs Processor Events queues Events Timeouts Commands runs ev1 Thread3 Thread4 Processor Events queues Events Timeouts Commands runs Processor Events queues Events Timeouts Commands runs Code, which runs in threads, organized in form of processors There are several processors in thread Processor process events from the thread event queues. In thread processors executed sequentially (according priority of coming events), therefore no locking required if two such processors communicate with each other. Threads also by request provide timeouts to the processors. Of course, processors can communicate (fire events) with processors from other threads. Such methods are fully thread-safe. Typically one extends such calls and provides more data with event id – buffers, commands, anything else. In such approach code in processors runs absolutely independent from each other and cannot block (in form of blocked mutexes). Any requests to other threads converted to the events and send to other. Other thread can send a response event. Another important problem is memory management. ev2 ev3

10 Memory management dabc::MemoryPool dabc::Buffer dabc::Buffer
References dabc::Buffer References Memory blocks dabc::Buffer References Concept of memory pools – pre-allocated blocks of memory. Once allocated, layout of memory pool can be fixed, which can avoid expensive (in terms of time) reallocation operation during run. One can reference different peaces of memory pool via buffer objects Many references on the same memory region allowed via reference counter. Scatter-gather lists are supported Only asynchronous (non-blocking) requests to memory pool are supported. Main aim – support of zero-copy concept across complete set of components Multiple references on resources (blocks) Copy of reference without copy of data Zero copy transfer: PCI/DMA → buffer → IB/DMA

11 User code - modules dabc::Module dabc::Module provides:
dabc::PoolHandle dabc::Timer dabc::Port dabc::Parameter dabc::Module provides: I/O ports for communications Pools handles to request memory Timers for timeouts processing Configuration & monitoring parameters Main place for user-specific code is module. Module has: i/o ports for communication with other components (or other modules) pools handles to request memory timers for timeout processing configuration parameters

12 Two kind of modules: sync & async
UserModule “Pool” “Input” “Output0” “Output1” Constructors mainly the same CreatePoolHandle("Pool", 2048, 1); CreateInput("Input", Pool(), 5); CreateOutput("Output0", Pool(), 5); CreateOutput("Output1", Pool(), 5); fCnt = 0; Explicit loop in ModuleSync Event processing in ModuleAsync Actual user code should be implemented as subclass of ModuleSync or ModuleAsync classes. Constructor of user module looks mostly the same for both kinds, difference is in implementing code. Sync: explicit main loop, linear code, blocks when requires resources (buffers), suited for simple communication patterns, requires explicit thread to run Async: event-based processing, more complex for implementation, more flexible in functionality, can share thread with other modules or devices Just look on simple example of user module, with one input and two outputs. The only task of such module is sending data consequently from input to both outputs. Probably, for such simple case explicit main loop can be a good solution, but if one starts to add more inputs or outputs and requires complex rules which input and when should deliver to some output, liner code becoming not so easy to implement. For complex tasks usually second approach of events processing with async module is used. That’s about data handling inside user code. Whats happens with buffers outside? void UserModule::MainLoop() { while (ModuleWorking()) { dabc::Buffer* buf = Recv(Input()); if (fCnt++ % 2 == 0) Send(Output(0), buf); else Send(Output(1), buf); } void UserModule::ProcessIOEvent(dabc::Port*) { while (Input()->CanRecv() && Output(fCnt % 2)->CanSend()) { dabc::Buffer* buf = Input()->Recv(); Output(fCnt++ % 2)->Send(buf); }

13 Transport & device – dataflow
node1 File UserModule “Pool” “Input” “Output0” “Output1” UserModule “Pool” “Input” “Output0” “Output1” Transport Transport Transport Transport Transport DataServer PCIeDevice NetworkDevice Transport: manages buffers queue runs in own thread decouples user code from actual transport functionality Device: represents hardware items manages several transports node2 UserModule “Pool” “Input” “Output0” “Output1” Transport connects ports between modules or with data source/drains. Device represents different hard- or software components, which can provide or consume data Transport connects device with module ports, one device can handle several transports Typical use is network I/O, file I/O, hardware board communications Transport manages its own buffer queue – such queue is sheared between modules if transport connect them locally. Typically transport runs its own thread, but it may also use same thread as module. Main aim of transport is to decouple user code from actual I/O code of framework Transport Transport NetworkDevice File

14 Many other features Central dabc::Manager class
Configuration with XML files Controlling interface and GUI dabc::Commands Factories (plugins) architecture Application and state machine Event building network (BNET) All this included in version 1 of DABC

15 Download via
DABC release 1.0 Download via DABC Controls Plugins Applications Core Slim (batch) DIM socket IB verbs* bnet mbs bnet-mbs ROC/udp ROC/ABB* core-test net-test bnet-test mbs bnet-mbs ROC Java GUI DABC itself is just small core library Useful functionality organized in form of plugins, which can be attached when they need There are already number of such plugins Socket handling with socket-specific thread and processor classes to implement event-based socket handling (via select() function) InfiniBand verbs plugin to utilize full functionality of that fast network technology with low CPU consumption Plugin for MBS (Multi Branch System), current standard DAQ in GSI. Provides set of useful components to connect to running DAQ, work with data formats, read/write data files Event building framework (bnet) – set of DABC-based components to implement NxM event building network, designed to decouple user data handling from actual networking Integration of CBM readout controller – data access either via Ethernet socket or via PCIx card with optical link to board. DABC will be used not only as DAQ for that board but also as device access library, which decouples user code from actual access mode All plugins just a libraries, which can be loaded by DABC at startup time. User can always select which libraries are required. Together with plugins there are number of applications, which are provided. It is either just a configuration file for some ready-to-use plugins (like MBS data combiner module) or implementation of subclasses with user code Plugins: Implementation of device/transports and experiment specific data handling code (programmers) Applications: Mainly setup or testing programs (users) ROC: CBM Readout Controller board ABB: Active buffer board (PCIe) IB: InfiniBand MBS: Multi Branch System (GSI DAQ) * external packages needed

16 DABC in CBM test beam roc::Readout STS GEM online go4 monitor Combiner
Sorter raw data calibrated data Input0 Input1 Input2 Output0 Output1 roc::Readout GEM First beam tests of CBM detector prototypes. First usage of DABC preliminary version for taking, sorting and storing data. online go4 monitor

17 DABC as access layer to ROC
ROC/udp plugin ROC/PCIe User access layer DABC Ethernet Users scripts, GUIs file I/O online monitor optic Readout controller can be connected either via Ethernet or by optic While DABC provides a lot of services as thread, memory, configuration management, It becomes a good basis for developing common access layer to the ROC functionality. As a result, same code used in tests setups and in DAQ applications, where many ROCs are connected PCIe

18 Planned CBM beam tests Readout boards Combiner boards Detectors FEE
Commodity PCs More test beams are planned In next beam it is planned to use many readout nodes and perform event building over IB network. Copper Optic InfiniBand

19 Event building network test
Readouts Senders InfiniBand Receivers Event builders Libs for all applications Defaults for all workers Explicit config for PCIe board Worker1 PCIe Rnd Worker2 Worker3 Rnd Controller Example of Bnet configuration of 4x4 net. Worker4 Rnd Rnd Rnd Rnd

20 Conclusion DABC is general-purpose DAQ framework
Can be used for different purposes – from small detectors tests to multi-nodes application Through its plugin architecture can be easy extended to specific needs Open for improvements and new ideas

21 Thanks!

Download ppt "First release of Data Acquisition Backbone Core"

Similar presentations

Ads by Google