1. First release of Data Acquisition Backbone Core
J. Adamczewski-Musch, H.G. Essel, N. Kurz, S. Linev
GSI Darmstadt, Germany
Experiment Electronics: Data processing group

2. FAIR accelerators FAIR baseline* SIS100 SIS300 p-linac HESR Super- FRS
p-bar target
p-linac
Super- FRS
SIS100
SIS300
HESR CR
RESR
Unilac
SIS 100
NESR
Antiproton Prod. Target
SIS18 Upgrade
SIS 300
Accelerator
SuperFRS
PANDA
Atomic Phys.
Plasma Phys.
Low Energy Exp.
High Energy Exp.
NESR Exp.
FLAIR
Atomic Physics
HADES & CBM
Experiment
FAIR baseline*
Facility for Antiproton and Ion Research
*FAIR Monthly, April 2009

3. CBM experiment at FAIR Transition Radiation Electro- Detectors
Dipole
magnet
Ring Imaging
Cherenkov
Detector
Transition
Radiation
Detectors
Resistive
Plate
Chambers
(TOF)
Electro-
magnetic
Calorimeter
Silicon
Tracking
System
Projectile
Spectator
(Calorimeter)
Micro Vertex
One of experiments in FAIR will CBM – Condensed Barion Matter experiment
Free-running electronic
Precise timing for all detectors

4. DAQ architectures Conventional DAQ CBM DAQ HLT HLT FLES
L1 trigger
HLT
HLT
FLES
CBM DAQ
Self-triggered Front-end all hits shipped to DAQ. Data push architecture
Detector
FEE buffer
Readout buffer
Switch
Processor farm
Storage
Fast links
Readout buffer outside radiation area. Many Gbyte storage easily possible. Allows L1 decision times of ms
High-throughput Event building
One cannot make filter (trigger) decision before complete event is reconstructed.
Event reconstruction based on time stamps (time slices building)
~1 TB/s of primary data rates up to event builders
First event selection done in processor farm.

5. InfiniBand performance studies
> 500 MBytes/s bidirectional
+ 25%
InfiniBand is probable candidate for use in event building.
Has low-latency and high bandwidth, implements zero-copy concept
Scales with number of nodes, low CPU usage
Meet CBM requirements
thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz

6. Hardware tests nXYTER, 128-channel self-triggered
readout chip with few ns time
resolution, DEFNI collaboration
Beside big plans for final system now we need to perform a lot of detectors tests
We think, that DAQ software components should be used also at that point for better integration into the final system.
For instance, we start investigation of self-triggered electronics with nXYTER chip.
The n-XYTER is a 128 channel asynchronous, self- triggered, self-sparcifying readout ASIC developed as a front-end for neutron scattering detector applications. In FAIR experiments like CBM it is used as prototype and test-bed of front-end electronics for several kinds of detectors prototypes.
The CBM readout controller (ROC), developed in Kirchhoff Institut für Physik, Heidelberg, is an FPGA-based board, designed to read time-stamped data from n-XYTER and transfer that data via optical link to PC. As alternative option, Ethernet can be used for data transfer.
CBM Readout controller (ROC),
Kirchhoff Institut für Physik, Heidelberg

7. Future DAQ requirements
Flexible – adopt different kinds of soft- and hardware
Compact – use only necessary code
Scalable – from small detector tests to 100-nodes clusters
Performing – low framework overhead
Multiprocessing & multithreading
I show only plans for CBM DAQ, but many other future experiments will demand for new features which are not provided by with our current DAQ system.
Therefore we start development of new framework, which should fulfill such a new requirements.
First problem which should be solved by DAQ is appropriate usage of multithreading capabilities of modern CPUs.

8. Multithreading Deadlock! Thread1 Thread2 runs runs Thread3 Thread4
void MainLoop() {
while (IsWorking()) { …
writeSomething(); }
runs
void MainLoop() {
while (IsWorking()) { …
doSomething(); }
runs
Deadlock!
Thread3
Thread4
void MainLoop() {
while (IsWorking()) { …
calcSomething(); }
runs
void MainLoop() {
while (IsWorking()) { …
readSomething(); }
runs
Now it is unavoidable to use multi-CPU/multi-core machines
Different tasks should run as different threads
Threads is just place of code, but not code itself
In simple case it is just kind of main loops, which running any code.
As long as such threads are not communicate with each other, there is no problems.
But if they should communicate, it often leads to different problems. For instance, dead locks.
To avoid them, you should really count locks, check their hierarchy and so on.
As a result, development of multithreading applications can become very complicated.
To avoid such problem in DABC, another approach was taken.

9. Multithreading in DABC
Processor
Events queues
Events
Timeouts
Commands
runs
Processor
Events queues
Events
Timeouts
Commands
runs
ev1
Thread3
Thread4
Processor
Events queues
Events
Timeouts
Commands
runs
Processor
Events queues
Events
Timeouts
Commands
runs
Code, which runs in threads, organized in form of processors
There are several processors in thread
Processor process events from the thread event queues.
In thread processors executed sequentially (according priority of coming events),
therefore no locking required if two such processors communicate with each other.
Threads also by request provide timeouts to the processors.
Of course, processors can communicate (fire events) with processors from other threads.
Such methods are fully thread-safe. Typically one extends such calls and provides more data with event id – buffers, commands, anything else.
In such approach code in processors runs absolutely independent from each other and cannot block (in form of blocked mutexes). Any requests to other threads converted to the events and send to other. Other thread can send a response event.
Another important problem is memory management.
ev2
ev3

10. Memory management dabc::MemoryPool dabc::Buffer dabc::Buffer
References
dabc::Buffer
References
Memory blocks
dabc::Buffer
References
Concept of memory pools – pre-allocated blocks of memory.
Once allocated, layout of memory pool can be fixed, which can avoid expensive (in terms of time) reallocation operation during run.
One can reference different peaces of memory pool via buffer objects
Many references on the same memory region allowed via reference counter.
Scatter-gather lists are supported
Only asynchronous (non-blocking) requests to memory pool are supported.
Main aim – support of zero-copy concept across complete set of components
Multiple references on resources (blocks)
Copy of reference without copy of data
Zero copy transfer: PCI/DMA → buffer → IB/DMA

11. User code - modules dabc::Module dabc::Module provides:
dabc::PoolHandle
dabc::Timer
dabc::Port
dabc::Parameter
dabc::Module provides:
I/O ports for communications
Pools handles to request memory
Timers for timeouts processing
Configuration & monitoring parameters
Main place for user-specific code is module.
Module has:
i/o ports for communication with other components (or other modules)
pools handles to request memory
timers for timeout processing
configuration parameters

12. Two kind of modules: sync & async
UserModule
“Pool”
“Input”
“Output0”
“Output1”
Constructors mainly the same
CreatePoolHandle("Pool", 2048, 1);
CreateInput("Input", Pool(), 5);
CreateOutput("Output0", Pool(), 5);
CreateOutput("Output1", Pool(), 5);
fCnt = 0;
Explicit loop in ModuleSync
Event processing in ModuleAsync
Actual user code should be implemented as subclass of ModuleSync or ModuleAsync classes.
Constructor of user module looks mostly the same for both kinds, difference is in implementing code.
Sync: explicit main loop, linear code, blocks when requires resources (buffers), suited for simple communication patterns, requires explicit thread to run
Async: event-based processing, more complex for implementation, more flexible in functionality, can share thread with other modules or devices
Just look on simple example of user module, with one input and two outputs. The only task of such module is sending data consequently from input to both outputs.
Probably, for such simple case explicit main loop can be a good solution, but if one starts to add more inputs or outputs and requires complex rules which input and when should deliver to some output, liner code becoming not so easy to implement. For complex tasks usually second approach of events processing with async module is used.
That’s about data handling inside user code. Whats happens with buffers outside?
void UserModule::MainLoop() {
while (ModuleWorking()) {
dabc::Buffer* buf = Recv(Input());
if (fCnt++ % 2 == 0) Send(Output(0), buf);
else Send(Output(1), buf); }
void UserModule::ProcessIOEvent(dabc::Port*) {
while (Input()->CanRecv() &&
Output(fCnt % 2)->CanSend()) {
dabc::Buffer* buf = Input()->Recv();
Output(fCnt++ % 2)->Send(buf); }

13. Transport & device – dataflow
node1
File
UserModule
“Pool”
“Input”
“Output0”
“Output1”
UserModule
“Pool”
“Input”
“Output0”
“Output1”
Transport
Transport
Transport
Transport
Transport
DataServer
PCIeDevice
NetworkDevice
Transport:
manages buffers queue
runs in own thread
decouples user code from actual transport functionality
Device:
represents hardware items
manages several transports
node2
UserModule
“Pool”
“Input”
“Output0”
“Output1”
Transport connects ports between modules or with data source/drains.
Device represents different hard- or software components, which can provide or consume data
Transport connects device with module ports, one device can handle several transports
Typical use is network I/O, file I/O, hardware board communications
Transport manages its own buffer queue – such queue is sheared between modules if transport connect them locally.
Typically transport runs its own thread, but it may also use same thread as module.
Main aim of transport is to decouple user code from actual I/O code of framework
Transport
Transport
NetworkDevice
File

14. Many other features Central dabc::Manager class
Configuration with XML files
Controlling interface and GUI
dabc::Commands
Factories (plugins) architecture
Application and state machine
Event building network (BNET)
All this included in version 1 of DABC

15. Download via http://dabc.gsi.de
DABC release 1.0
Download via
DABC
Controls
Plugins
Applications
Core
Slim
(batch)
DIM
socket
IB verbs*
bnet
mbs
bnet-mbs
ROC/udp
ROC/ABB*
core-test
net-test
bnet-test
mbs
bnet-mbs
ROC
Java GUI
DABC itself is just small core library
Useful functionality organized in form of plugins, which can be attached when they need
There are already number of such plugins
Socket handling with socket-specific thread and processor classes to implement event-based socket handling (via select() function)
InfiniBand verbs plugin to utilize full functionality of that fast network technology with low CPU consumption
Plugin for MBS (Multi Branch System), current standard DAQ in GSI. Provides set of useful components to connect to running DAQ, work with data formats, read/write data files
Event building framework (bnet) – set of DABC-based components to implement NxM event building network, designed to decouple user data handling from actual networking
Integration of CBM readout controller – data access either via Ethernet socket or via PCIx card with optical link to board. DABC will be used not only as DAQ for that board but also as device access library, which decouples user code from actual access mode
All plugins just a libraries, which can be loaded by DABC at startup time.
User can always select which libraries are required.
Together with plugins there are number of applications, which are provided.
It is either just a configuration file for some ready-to-use plugins (like MBS data combiner module) or implementation of subclasses with user code
Plugins: Implementation of device/transports and experiment specific data handling code (programmers)
Applications: Mainly setup or testing programs (users)
ROC: CBM Readout Controller board
ABB: Active buffer board (PCIe)
IB: InfiniBand
MBS: Multi Branch System (GSI DAQ)
* external packages needed

16. DABC in CBM test beam roc::Readout STS GEM online go4 monitor Combiner
Sorter
raw data
calibrated data
Input0
Input1
Input2
Output0
Output1
roc::Readout
GEM
First beam tests of CBM detector prototypes.
First usage of DABC preliminary version for taking, sorting and storing data.
online go4 monitor

17. DABC as access layer to ROC
ROC/udp
plugin
ROC/PCIe
User access layer
DABC
Ethernet
Users scripts,
GUIs
file I/O
online monitor
optic
Readout controller can be connected either via Ethernet or by optic
While DABC provides a lot of services as thread, memory, configuration management,
It becomes a good basis for developing common access layer to the ROC functionality.
As a result, same code used in tests setups and in DAQ applications, where many ROCs are connected
PCIe

18. Planned CBM beam tests Readout boards Combiner boards Detectors FEE
Commodity PCs
More test beams are planned
In next beam it is planned to use many readout nodes and perform event building over IB network.
Copper
Optic
InfiniBand

19. Event building network test
Readouts
Senders
InfiniBand
Receivers
Event
builders
Libs for all applications
Defaults for all workers
Explicit config for PCIe board
Worker1
PCIe
Rnd
Worker2
Worker3
Rnd
Controller
Example of Bnet configuration of 4x4 net.
Worker4
Rnd
Rnd
Rnd
Rnd

20. Conclusion DABC is general-purpose DAQ framework
Can be used for different purposes – from small detectors tests to multi-nodes application
Through its plugin architecture can be easy extended to specific needs
Open for improvements and new ideas

21. Thanks!