NA49-future Meeting, January 26, 20071Ervin Dénes, KFKI - RMKI DATE the DAQ s/w for ALICE (Birmingham, Budapest, CERN, Istanbul, Mexico, Split, Zagreb collaboration) and its possible application at NA49-future

NA49-future Meeting, January 26, 20072Ervin Dénes, KFKI - RMKI ALICE Experiment Counting rooms Cavern Computing Center DDLs 10Gb Ethernet Machine: CERN, Point 2 ALICE detectors: 17 Online Systems: ECS, DAQ, TRG, HLT, DCS Readout: ~ Mbps Storage rate: max GB/s Events: 2.5 MB (p-p), 87 MB (Pb-Pb) DAQ Software: DATE V5 DAQ

NA49-future Meeting, January 26, 20073Ervin Dénes, KFKI - RMKI GDC ALICE DAQ architecture CTP LTU TTC FERO LTU TTC FERO LDC BUSY Rare/All Event Fragment Sub-event Event File Storage Network TDS PDS L0, L1a, L2 262 DDLs EDM LDC Load Bal. LDC HLT Farm FEP DDL H-RORC 10 DDLs 10 D-RORC 10 HLT LDC 123 DDLs TDS DS Event Building Network 329 D-RORC 175 Detector LDC 50 GDC 25 TDS 5 DSS

NA49-future Meeting, January 26, 20074Ervin Dénes, KFKI - RMKI Trigger Level 0,1 Trigger Level 2 High-Level Trigger Decision Data ALICE online logical model Detector Digitizers Front-end Pipeline/Buffer Decision Readout Buffer Decision Subevent Buffer Event Buffer Storage Local Data Concentrator (LDC) Global Data Collector (GDC) Data transfer Detector Data Link (DDL) DAQ Read-Out Receiver Card (D-RORC) ALICE Terminology Transient Data Storage (TDS) Permanent Data Storage (PDS) Decision Event Building

NA49-future Meeting, January 26, 20075Ervin Dénes, KFKI - RMKI ALICE Detector Data Link

NA49-future Meeting, January 26, 20076Ervin Dénes, KFKI - RMKI LDC DDL architecture PCI Bus Front-End Read-Out DAQ Read-out Receiver Card (D-RORC) Source Interface Unit Forward Channel (Raw data) Backward Channel (Pedestals, control) Destination Interface Unit Detector Data Link (DDL) : - Source Interface Unit - Transmission media - Destination Interface Unit Standard detector/DAQ interface 100 MHz 64-bit PCI card

NA49-future Meeting, January 26, 20077Ervin Dénes, KFKI - RMKI DAQ Read-Out Receiver Cards (D-RORC) PCI-X adapter for 2 DDL PCI master: autonomous DMA

NA49-future Meeting, January 26, 20078Ervin Dénes, KFKI - RMKI Readout System Performance Motherboard with dual Xeon 2.4 GHz Six PCI-X slots, 4 bus segments ( ), 2 controllers Linux OS ALICE Data-Acquisition software (DATE)

NA49-future Meeting, January 26, 20079Ervin Dénes, KFKI - RMKI Performance: 6 D-RORCs Testing the fully populated PC using data source internal to PCI interface Interoperability test Measure the maximal input bandwidth PCI #6 PCI #5 PCI #4 PCI #3 PCI #2 PCI #1 Segment #1 #2 #3 #4 Controller #1 #2 1 Ch Aggregate Bandwidth [MB/s] Normalized Bandwidth [MB/s/Ch]

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DAQ software DDL softwarewe can freely use DAQ framework (DATE)does we need it? Performance Monitoring (AFFAIR) Data quality monitoring (MOOD) Information dissemination Documentation: 4 DAQ sw packages + ECS: fully documented. User’s guide released and printed Regular DAQ trainings (70 people in last 3 years) Linux Linux SLC4 now. Process defined to produce, distribute, install new versions Code management system: CVS. Release packaging and distribution: RPM (~20 MB) Automatic installation of Linux and DATE on DAQ nodes

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKIRORC Memory Management’s Principle Firmware data length transfer status Ready FIFO Transfer status possible values: ffffffff ffffffff unloaded (set by sw) loaded, no DTSTW (set by fw) else else loaded, DTSTW (set by fw) Free FIFO block size Index of Ready FIFO start address Page aligned continuous user memory outside Linux’s memory space, reachable for DMA and user as well. (physmem module necessary) Free blocks for events or event fragments Free FIFO / Ready FIFO depth: 128 Maximum block size: 2Mbyte

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI The Free FIFO D-RORC PC memory bank Firmware readout page address Free FIFO PC CPU Allocation of free pages

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Direct Memory Access D-RORC Firmware PC memory bank DDL No involvement PC CPU

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI The Ready FIFO D-RORC PC memory bank readout DDL Ready FIFO Firmware addresspage status addresspage status addresspage status Delivery of filled pages PC CPU

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DDL Software All functions accessible as interactive commands or API Script-based interpreter for sequence of operations: Sending command to the FEE Reading FEE status printing the status comparing the status polling the status Downloading data into the FEE from a file Reading data from the FEE writing data into a file comparing data with data in a file TPC configuration: < 0.3 s DDL performance Parallelism FERO DDL definepedestal_addr0x1FFF defineenable_pedestal0x2C resetSIU write_commandenable_pedestal write_blockpedestal_addr pedestal.hex%x read_and_check_blockpedestal_addrpedestal.hex%x definepedestal_addr0x1FFF defineenable_pedestal0x2C resetSIU write_commandenable_pedestal write_blockpedestal_addr pedestal.hex%x read_and_check_blockpedestal_addrpedestal.hex%x D-RORC LDC

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DATE Run Control Control Configuration Main parameters Display SMI State of all nodes Main counters of all nodes Compatible with ALICE ECS

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DATE Configuration Database Database content DATE Roles Actors of DATE system: LDCs, GDCs… Trigger Trigger masks Detectors Front-end equipment of LDCs Event building control Event building rules Banks Memory banks to operate DATE

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DATE Infologger (1/2) Online view Online selection of visible fields Online selection of message displayed

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DATE Infologger (2/2) Offline view Queries in the message archives Selection of visible fields and of message displayed

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Data quality monitoring: MOOD MOOD: Monitoring Of Online Data DATE + ROOT environments MOOD framework Interfaces to detector code Applications: Raw data integrity Detector performance

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Performance monitoring: AFFAIR System view / Aggregated performances Event building bandwidth Event numbers Individual nodes view CPU usage Input/Ouput Node status

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Event Building Switch ALICE baseline: TCP/IP over switched Ethernet Computing Data Challenges Event-Building Switch CERN frame contracts Needs of IT and experiments Good prices On site maintenance (company+IT) Force 10 Model 1200

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Qualification of event-building switch

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Storage Arrays Performance Local disk buffer at Pt2 Storage network: Fibre Channel currently FC 2G or 4G (Brocade,QLogic) 16 ports switches with 4 uplinks for interconnect

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Storage Arrays Performance Transient Data Storage Storage arrays (e.g. Infortrend A16F 2 FC 2G ports, 16 SATA II HD) Performance for a RAID 5 set of 5 disks

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DDL and DATE for NA49-future? (1/2) Data transfer DDL and D-RORC produced and used for longtime Match speed requirements Few (< 5) PCs are enough for the task We have the knowledge how to implement to NA49-future About 1 man-year of work Problems with sub-detectors using VME In ALICE exist VME boards where DDL SIU cards are attached, but interfaces are specific to the given ALICE sub-detector Or the design VME – DDL interface is necessary for TOF and CAMACs Interconnect with the trigger system Trigger “box” has to be designed

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DDL and DATE for NA49-future? (2/2) DAQ software Software (DATE V5) released and documented. Process defined to produce, distribute, install new versions. Everything (ECS, monitoring, etc.) included Can we get it from ALICE? They give it only with support They can not support us during this year (integration with 17 ALICE sub-detectors) Modifications necessary if TOF and CAMACs are used without the DDL Is it shooting at sparrows with a high caliber gun ? DAQ fabric hardware Event building switch is necessary Simple storage network has to be added

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Thank you

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI Detector Data Link (DDL) Detector readout: fast data transfer to PC memory Electronics configuration: pedestals download Interface and data-transfer detector/DAQ Radiation tolerant card All components are radiation tolerant including FPGA (ACTEL) 10 year doses in ALICE (at TPC inner radius): total ionising dose1.6 krad neutron fluence3.9·10 11 cm -2 charged hadron fluence8·10 9 cm -2 ACTEL ProASIC adopted as baseline

NA49-future Meeting, January 26, Ervin Dénes, KFKI - RMKI DATE V5 Software ALICE Data Acquisition and Test Environment (DATE): software framework for the ALICE DAQ system  Latest DATE version V5:  DATE V5.x kits: RPM package (~20 MB), User ’ s Guide (~470 pages)  Platforms: IA32 family, SLC4 using a 2.6 kernel  Development: ~20 packages, mostly in C, using CVS  Auxiliary Software: DIM, SMI, MySQL, Tcl/Tk, libshift, ROOT, VMEbus driver  Testing: reference system, on-line data challenges, test beams DATE V5 Packages