Presentation is loading. Please wait.

Presentation is loading. Please wait.

D ata A cquisition B ackbone C ore 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU.

Published byPierce Peters Modified over 8 years ago

Similar presentations

Presentation on theme: "D ata A cquisition B ackbone C ore 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU."— Presentation transcript:

1 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU RP6 project JRA1 FutureDAQ RII3-CT-2004-506078 Data Acquisition Backbone Core J.Adamczewski, H.G.Essel, N.Kurz, S.Linev GSI, Experiment Electronics, Data Processing group Motivation Data-flow engine, control Event building network Performance To do

2 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 2 CBM data acquisition data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher switching network BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution W.F.J.Müller, 2004

3 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 3 CBM DAQ features summary Self-triggered time stamped data channels. Complex trigger algorithms  transport until filter farm FPGA controlled data flows Event building on full data rate ~1TB/s Event builder network BNet: ~1000 nodes, high-speed interconnections Linux may run on most DAQ nodes (even FPGAs)

4 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 4 PCIe FE Front end board: sampling ADCs, clock distribution Active Buffer board *: PCI express card PC ABB PCIe 2.5 GBit/s bi-directional (optical) link: data, clock Use case example: Frontend components test The goal: Detector tests FEE tests Data flow tests FE: Frontend board ABB: Active Buffer board * A.Kugel, G.Marcus, W.Gao, Mannheim University Informatics V

5 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 5 GE switch PC ABB PCIe DCB FE IB switch FE DC Front end board: sampling ADCs, clock distribution Data combiner boards, clock distribution to FE Active Buffer board: PCI express card 8-20 PCs dual/quad PC ABB PCIe 88 44 Scales up to 10k channels, 160 CPUs ~625 Mb/s bi-directional (optical) link: data, clock ~2.5 Gb/s data links The goal: Investigate critical technology Detector tests Replace existing DAQ FE: Frontend board DC: Data combiner board ABB: Active Buffer board GE: Gigabit Ethernet IB: InfiniBand MBS: Multi Branch System MBS Use case example: middle-size setup

6 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 6 Driving forces and motivation for DABC 2004 → EU RP6 project JRA1 FutureDAQ* 2004 → CBM FutureDAQ for FAIR 2005 → FOPI DAQ upgrade (skipped) 2007 → NUSTAR DAQ * RII3-CT-2004-506078 1996 → MBS future 50 installations at GSI, 50 external http://daq.gsi.de Detector tests FE equipment tests Data transport Time distribution Switched event building Software evaluation MBS event builder General purpose DAQ Data Acquisition Backbone Core Intermediate demonstrator Requirements connect (nearly) any front-ends handle triggered or self-trigger front-ends process time stamped data streams build events over fast networks provide data flow control (to front-ends) interfaces to plug in application codes connect MBS readout or collector nodes be controllable by several controls frameworks

7 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 7 DABC key components Data-flow engine –memory and buffers management –threads and events management –data processing modules with ports, parameters, timers –transport and device classes, file I/O –back pressure mechanism Slow control and configuration –components setup on each node –parameters monitoring/changing –commands execution –state machine logic –user interface

8 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 8 Data flow engine DABC Module port DABC Module port process Local transport A module processes data of one or several data streams. Data streams propagated through ports, which are connected by transports Queue

9 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 9 Data flow engine DABC Module port DABC Module port process Device Net transport Device Net transport Queue Network A module processes data of one or several data streams. Data streams propagated through ports, which are connected by transports and devices

10 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 10 Memory management Main features: All memory to be used in modules for transport organized in memory pools Memory pool consists of one or several blocks of memory, divided on equal peaces - buffers Each buffer in memory pool can be referenced once for writing and any times for reading Several references may be combined in gather list Use for transport: Only data from memory pools can be transported via ports Each module port associated with only memory pool Transport between two modules in same application done via pointer Zero-copy network transport where supported (InfiniBand) Support of gather lists for all kind of transports

11 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 11 Threads and event management DABC Module A processInput processCommand Event manager Thread with modules Device thread Commands are synchronized with the data flow also through the Event manager Data ready event A device, i.e. socket device controls several ports (transports). Once a queue buffer is filled, the transport signals the Event manager, which in turn calls the processInput function of the associated module. Transport Queue DABC Module B Manager thread Command event

12 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 12 Back pressure mechanism Basic idea: sender allowed to send packets only after receiver confirms (with special acknowledge message) that it has enough resources to receive these packets Implemented on transport layer (not visible for user) Impact on module code: No additional efforts is required Can be enabled/disabled for any port To block connection - just do not read packets from it Pro: easy method for traffic control in small system Con: easy way to block complete network when single node is hanging

13 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 13 Class diagram as currently implemented Socket InfiniBand PCIboard Port Bnet Module

14 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 14 Slow control Tools for control Configuration via XML files State machines, Infospace, message/error loggers, monitoring Communication: Webserver, SOAP, DIM Connectivity through DIM to: LabView, EPICS, Java, any DIM client/server Java with NetBeans (Matisse GUI builder) maybe soon in Eclipse Front-end controls? Mix of cooperating control systems First LabView and Java GUIs operable

15 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 15 Controls & monitoring communication XDAQ Executive (process, address space) Web server (SOAP) DIM client: Java GUI LabView GUI EPICS GUI SOAP client: Java GUI Infospace* XDAQ Application Modules, command queue DABC data-flow DIM server State machine Web browser GRIDCC * Infospace – remotely accessible parameters

16 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 16 LabView-DIM Control GUI* Generic construction of fixed parameter table from DIM servers * Dietrich Beck, EE

17 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 17 Java-DIM Control Generic construction from commands and parameters offered by DABC DIM servers Applications create the commands and parameters and publish Ratemeter, trending, statistics

18 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 18 Generic Java DIM GUI controls

19 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 19 Event building network (BNet) data dispatcher FEE deliver time stamped data CNet collect data into buffers Detector collect ~50000 FEE chips event dispatcher switching network BNet sort time stamped data ~1000 links a 1 GB/sec HNet high level selection to high level computing and archiving ~1 GB/sec Output processing PNet process events level 1&2 selection subfarm ~100 subfarms ~100 nodes per subfarm ~10 dispatchers → subfarm ~1000 collectors ~1000 active buffers TNet time distribution

20 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 20 collecting sorting tagging building filtering analysis collecting sorting tagging building filtering analysis IB Linux GE analysis archive Linux GE: Gigabit Ethernet IB: InfiniBand frontend DataDispatcher frontend other frontend MBS readout Sender BNet dataflow – bidirectional approach DABC Receiver

21 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 21 plugin input BNet – modules view plugin* Readout plugin SubeventCombiner DataReceiver DataSender Network NxM plugin EventBuilder plugin* EventFilter M sender nodes 6 different modules N receiver nodes N outputs M inputs Analisys / Storage * optional

22 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 22 BNet components BNet modules: Readout – collecting packets from readout channels (MBS, PCI-board, other) SubeventCombiner – search and combines data, belonging to same event or time frame DataSender – sends subevent data over network to correspondent event building node DataReceiver – forwards packets from network to event builder EventBuilder – build complete event EventFilter – performs first event filtering BNet user plugins: finds data tags (event id or time stamp) in raw packet resort data for sending over net build event from received subevent packets implements filtering algorithm for events

23 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 23 InfiniBand – testbench for BNet InfiniBand - reliable low-latency zero-copy high-speed data transport Aim: Prove of principal as event building network candidate for CBM Tests last year: GSI cluster - 4 nodes, SDR Forschungszentrum Karlsruhe* (March 2007) – 23 nodes DDR UNI Mainz** (August 2007) - 110 nodes DDR DDR – double data rate (up to 20 Gb/s), SDR – single data rate (10 Gb/s) SDR (GSI)DDR (Mainz) Unidirectional0.98 GB/s1.65 GB/s Bidirectional0.95 GB/s1.3 GB/s Point-to-point tests Rate per node Gb Ethernet~100 MB/s InfiniBand~800 MB/s BNet prototype (GSI, 4 Nodes) * thanks to Frank Schmitz, Ivan Kondov and Project CampusGrid in FZK ** thanks to Klaus Merle and Markus Tacke at the Zentrum für Datenverarbeitung in Uni Mainz

24 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 24 Scaling of asynchronous traffic

25 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 25 Chaotic (async.) versus scheduled (sync.)

26 D ata A cquisition B ackbone C ore http://www-linux.gsi.de/~dabc/dabc/DABC.php 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 26 DABC further tasks Achieved: Control infrastructure –setup, configure –communication –very first Java GUI Data flow engine –multi-threading –InfiniBand –Sockets (Gigabit Ethernet) –first PCIexpress board –good performance –back pressure To do: Data formats Error handling/recovery MBS event building Time-stamped data Final API definitions Documentation

Download ppt "D ata A cquisition B ackbone C ore 27.09.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported by EU."

Similar presentations

Network II.5 simulator ..

Network II.5 simulator ..
NSS 2008, Dresden, High Performance Event Building over InfiniBand Networks GSI Helmholtzzentrum für Schwerionenforschung GmbH 20.10.081.

NSS 2008, Dresden, High Performance Event Building over InfiniBand Networks GSI Helmholtzzentrum für Schwerionenforschung GmbH
First release of Data Acquisition Backbone Core

First release of Data Acquisition Backbone Core
Yokogawa Network Solutions Presents:

Yokogawa Network Solutions Presents:
FEE/DAQ Demonstrator Walter F.J. Müller, GSI, Darmstadt for the CBM Collaboration 3 rd FutureDAQ Workshop GSI, Darmstadt, October 11, 2005.

FEE/DAQ Demonstrator Walter F.J. Müller, GSI, Darmstadt for the CBM Collaboration 3 rd FutureDAQ Workshop GSI, Darmstadt, October 11, 2005.
CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/20 New Experiences with the ALICE High Level Trigger Data Transport.

CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/20 New Experiences with the ALICE High Level Trigger Data Transport.
CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/27 A Control Software for the ALICE High Level Trigger Timm.

CHEP04 - Interlaken - Sep. 27th - Oct. 1st 2004T. M. Steinbeck for the Alice Collaboration1/27 A Control Software for the ALICE High Level Trigger Timm.
DABCDABC D ata A cquisition B ackbone C ore RT2010 J.Adamczewski-Musch, H.G.Essel, S.Linev 1 Data Acquisition Backbone Core Framework.

DABCDABC D ata A cquisition B ackbone C ore RT2010 J.Adamczewski-Musch, H.G.Essel, S.Linev 1 Data Acquisition Backbone Core Framework.
Data Acquisition Backbone Core DABC J. Adamczewski, H.G. Essel, N. Kurz, S. Linev GSI, Darmstadt The new Facility for Antiproton and Ion Research at GSI.

Data Acquisition Backbone Core DABC J. Adamczewski, H.G. Essel, N. Kurz, S. Linev GSI, Darmstadt The new Facility for Antiproton and Ion Research at GSI.
Artdaq Introduction artdaq is a toolkit for creating the event building and filtering portions of a DAQ. A set of ready-to-use components along with hooks.

Artdaq Introduction artdaq is a toolkit for creating the event building and filtering portions of a DAQ. A set of ready-to-use components along with hooks.
A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research.

A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research.
DLS Digital Controller Tony Dobbing Head of Power Supplies Group.

DLS Digital Controller Tony Dobbing Head of Power Supplies Group.
Boosting Event Building Performance Using Infiniband FDR for CMS Upgrade Andrew Forrest – CERN (PH/CMD) Technology and Instrumentation in Particle Physics.

Boosting Event Building Performance Using Infiniband FDR for CMS Upgrade Andrew Forrest – CERN (PH/CMD) Technology and Instrumentation in Particle Physics.
TRIGGER-LESS AND RECONFIGURABLE DATA ACQUISITION SYSTEM FOR POSITRON EMISSION TOMOGRAPHY Grzegorz Korcyl 2013.

TRIGGER-LESS AND RECONFIGURABLE DATA ACQUISITION SYSTEM FOR POSITRON EMISSION TOMOGRAPHY Grzegorz Korcyl 2013.
ETRAX CRIS architecture and Xilinx FPGA Peter Zumbruch Experiment control systems group GSI (KS/EE)

ETRAX CRIS architecture and Xilinx FPGA Peter Zumbruch Experiment control systems group GSI (KS/EE)
GBT Interface Card for a Linux Computer Carson Teale 1.

GBT Interface Card for a Linux Computer Carson Teale 1.
C.Combaret, L.Mirabito Lab & beamtest DAQ with XDAQ tools.

C.Combaret, L.Mirabito Lab & beamtest DAQ with XDAQ tools.
DABCDABC D ata A cquisition B ackbone C ore 12.12.07 J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported.

DABCDABC D ata A cquisition B ackbone C ore J.Adamczewski, H.G.Essel, N.Kurz, S.Linev 1 Work supported.
Understanding Data Acquisition System for N- XYTER.

Understanding Data Acquisition System for N- XYTER.
6-10 Oct 2002GREX 2002, Pisa D. Verkindt, LAPP 1 Virgo Data Acquisition D. Verkindt, LAPP DAQ Purpose DAQ Architecture Data Acquisition examples Connection.

6-10 Oct 2002GREX 2002, Pisa D. Verkindt, LAPP 1 Virgo Data Acquisition D. Verkindt, LAPP DAQ Purpose DAQ Architecture Data Acquisition examples Connection.

Similar presentations

Ads by Google