1. Developments and applications of DAQ framework DABC v2 http://dabc.gsi.de Jörn Adamczewski-Musch, Nikolaus Kurz, Sergey Linev GSI / Experiment Electronic CHEP 14.04.2015

2. DABC core functionality  Multi-thread, multi-node applications  Zero-copy transport approach full support for InfiniBand VERBS advanced sockets treatment with select()  Flexible plugin interface MBS, HADAQ, ROOT, Go4, EPICS, DIM, FESA  http for slow control JavaScript ROOT as web GUI  In development since 2007 v1.0 in 2009, now v2.7 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de2

3. Local communication 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de3 ReceiverSender memory pool buffer File, socket, PCIe,... Consumer  Pre-allocated memory pool  Reference counter for each buffer  Buffers queue between modules  No memcpy between sender and receiver queue

4. Remote communication 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de4 Transport Sender memory pool buffer Producer Receiver Transport memory pool buffer Consumer node1 node2  Same approach as in local  No memcpy for InfiniBand  Different communication patterns: 1  1, 1  N, N  1, N x M queue

5. InfiniBand tests with DABC  ½ fat tree topology  ~800 nodes with QDR IB  12 spine and 34 leaf switches 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de5 ibspine01 ibswitch01 0102030405060708 ibswitch07 4950515253545556 ibswitch02 0910111213141516 ibswitch03 1718192021222324 ibswitch04 2526272829303132 ibswitch05 3334353637383940 ibswitch08 5758596061626364 ibswitch09 6566676869707172 ibswitch10 7374757677787980 ibswitch11 8182838485868788 ibspine02ibspine03ibspine04 ibswitch06 4142434445464748 ibswitch12 8990919293949596  all-to-all traffic pattern  1.25 TB/s or 80% of available bandwidth Results from May 2011

6. PEXOR/KINPEX plug-ins 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de6 PEXOR (KINPEX): Pci-EXpress Optical Receiver Lattice (Kintex) FPGA 4 lane PCIe 4 high speed SFP connectors FEBEX (ADC), POLAND (QFW), TAMEX (TDC) various frontends Detector GOSIP protocol, optical chains pexorplugin::Device InitDaq() UserReadout() RequestToken() StartAcquisition()... poland::Device InitDaq() UserReadout() explodertest::Device InitDaq() UserReadout() subclasses DABC hardware libpexor.so user library pexor.ko Linux kernel module gosipcmd control tool

7. PEXOR/KINPEX plug-ins 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de7  Use generic pexor kernel module and library  Device base class: common functionality (gosip, DMA, interrupts) interface methods (InitDAQ(), UserReadout() )  Device subclasses: Frontend specific, may implement “user” code  Readout modes: Free running (polling request) Request after trigger interrupt (emulates MBS DAQ) “automatic” within trigger interrupt handler http://web-docs.gsi.de/~adamczew/dabcpexor/index.html

8. http::Server in DABC  Access to different data in DAQ application  JavaScript ROOT as user interface  One could integrate and access data from different sources: DABC, ROOT, Go4, MBS, EPICS, DIM, FESA,...  Same data could be accessed by Go4 and displayed using native ROOT graphics  One also could use command-line tools to access DAQ/analysis data via http or socket 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de8

9. Heterogeneous distributed applications 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de9 MBS DAQ shell> mbs -dabc mbs> @startup mbs> open file test.lmd mbs> sta acq cmd Go4 analysis linev@lxg0538:~ $ go4analysis -stream localhost GO4- *> Reading library: libGo4UserAnalysis GO4-*> Close of TUserSource GO4-*> Create default analysis with processor class TFirstStepProcessor GO4-*> Use class TStreamEvent as output event GO4-*> Welcome to Go4 Analysis Framework Release v4.6.3 (build 40603) ! GO4-*> Analysis: Added analysis step Analysis GO4-*> Main: starting analysis in batch mode... GO4-!> Fail to open AutoSave file Go4AutoSave.root GO4-*> Analysis LoadObjects: Failed to load from file Go4AutoSave.root f_evt_get_open for STREAM: port=6002 timeout=1 GO4-*> Factory: Create input event for MBS GO4-*> Event MbsEvent101 has source localhost class: TGo4MbsStream GO4-*> Factory: Create event processor Processor GO4-*> Create TFirstStepProcessor Processor GO4-*> Executing ROOT script first.C GO4-*> Factory: Create output event OutputEvent GO4-*> Event OutputEvent has source Processor class: TFirstStepProcessor GO4-*> Analysis -- Initializing EventClasses done. TGo4Sniffer socket Publisher HADAQ Publisher socket EPICS DABC DAQ DABC master dabc::Publisher collects data from remote build global hierarchies version control http socket mbs plugin socket DABC agents log status

10. MBS/GOSIP web gui 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de10

11. DABC/PEXOR/GOSIP web gui 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de11

12. POLAND frontend web gui 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de12

13. HADES detector HADES is a versatile detector for a precise spectroscopy of e + e - pairs (di-electrons) and charged hadrons produced in proton, pion and heavy ion induced reactions in a 1÷3.5 GeV kinetic beam energy region 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de13 http://www-hades.gsi.de http://blogs.helmholtz.de

14. IOC HADES DAQ before 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de14 TRB small hld hld IOC EPICS GUI Event builder server /shm HYDRA2  ~30 TRB frontend boards  ~10 KHz, max 300 MB/s  simple round-robin approach  C-based, single thread code  EPICS IOCs for DAQ slow control  file-based quality monitoring UDP client

15. DABC HADES DAQ now 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de15 TRB hld Combiner output stream  DABC on all event builders  All infrastructure remains as is  Socket data stream to online analysis  Used in production beams 2014 http DABC hld Combiner output stream http DABC hld Combiner output stream http DABC hld Combiner output stream http EPICS GUI /shm server client HYDRA2 UDP

16. HADES DAQ upgrade  New RICH and ECAL electronic more than 100 TRBs (30 up to now)  Use in many places FPGA TDCs need for special calibration procedure  Potentially free-running readout for RICH need to produce subevents afterwards  HLD data format should be preserved 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de16

17. Change networking topology 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de17 TRB DABC TRB DABC 10Gb Ethernet All data from single TRB always send to same DAQ node Every DAQ node can perform TDC calibration and event selection immediately Event building done afterwards with TCP/IP or IB UDP TCP

18. Next plans  FPGA TDCs calibrations in DABC already done, including web control GUI  Event building network (BNet) for HADES planned as v3 release, end of 2015  Accommodate C++11 features native threads, references probably as v4 in ~2017 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de18

19. Conclusion  Flexible framework for different DAQ tasks  Web and JSROOT as user interface  Used in different test beams  Production DAQ for HADES experiment 14.4.2015Sergey Linev, DABC v2, dabc.gsi.de19