Philippe Vannerem CERN / EP ICALEPCS - Oct03 Distributed Control and Monitoring of High-Level Trigger Processes on the LHCb on-line Farm Philippe Vannerem CERN / EP ICALEPCS - Oct03

Contents the Experiment Control System of LHCb the Event Filter Farm GAUDI, the software framework for event processing GAUCHO, how to glue GAUDI to SCADA Communicating histograms Conclusions ICALEPCS / Oct-03 Philippe Vannerem

ECS Scope DCS Devices (HV, LV, GAS, Temperatures, etc.) Detector Channels Trigger Front End Electronics Experiment Control System Readout Network Processing/Filtering Storage DAQ External Systems (LHC, Technical Services, Safety, etc)

ECS: Generic SW Architecture T.S. LHC DCS DAQ DSS ... ... GAS DetDcs1 DetDcsN DetDaq1 Status & Alarms Commands SubSys1 SubSys2 SubSysN Dev1 Dev2 Dev3 DevN To Devices (Hardware or Software) Distributed Hierarchical Control Commercial components and industrial standards ICALEPCS / Oct-03 Philippe Vannerem

ECS: Hierarchical control Each node is able to control and monitor its subtrees: Propagate commands to child level Summarize information for the parent level Implement specific behaviour & Take local decisions Sequence & Automate operations Recover errors Partitioning Exclude one or more of its subtrees which then can run stand-alone User Interfacing Present information and receive commands Filter and display alarms DCS Tracker Muon HV Temp HV GAS ICALEPCS / Oct-03 Philippe Vannerem

ECS: implementation Implementation in commercial SCADA system software: PVSS II device description + access (OPC, Profibus, drivers) alarm handling+display user interface builder + control scripting archiving Joint COntrols Project between the LHC experiments: The Framework: hierarchical control templates + tools and control modules for common HW in PVSS Important building block: SMI++: State Management Interface, distributed control Abstract behaviour modeling Finite State Machines Automation & Error recovery with a Rule based system LHCb ECS is based the JCOP Framework and will extend it for its specific needs. ICALEPCS / Oct-03 Philippe Vannerem

LHCb’s DAQ&trigger architecture Level-1 Traffic HLT Traffic Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE TRM 126 Links 1100 / 25 = 44 kHz 5.5 GB/s 323 Links 40 / 10 = 4 kHz 1.6 GB/s Multiplexing Layer Switch Switch Switch Switch Switch 64 Links 32 Links Readout Network L1-Decision Sorter TFC System 94 Links 7.1 GB/s Storage System … 94 SFCs SFC Switch CPU SFC Switch CPU SFC Switch CPU SFC Switch CPU CPU Farm Gb Ethernet Level-1 Traffic Mixed Traffic HLT Traffic ~1800 CPUs

Event Filter Farm Control EFF: ~1800 commodity rack-mounted CPUs CPUs+SFCs linked to ECS PCs in separate Ethernet LAN ECS Tasks: Hardware control (remote boot, reset,…) + monitoring (CPU usage, memory, IO, etc.) Software control + monitoring Level-1+ High-Level Trigger algorithms are implemented using the GAUDI framework Event Builder Switch SFC CPU . . . . ICALEPCS / Oct-03 Philippe Vannerem

GAUDI Framework: design choices Separation between data and algorithms Separation between “transient” and “persistent” data Generic component interfaces OO Framework implemented with C++, runs on Win2000+Linux ICALEPCS / Oct-03 Philippe Vannerem

GAUDI: component architecture Interface defined in C++ pure virtual class User specifies the components to be used in the Joboptions file DLLs are loaded at run-time Algorithms can be configured in the joboptions file ICALEPCS / Oct-03 Philippe Vannerem

Interface Requirements Real-time system to use on the EFF Interface to the trigger algorithms written in Gaudi event data processing framework. Interface to the SCADA-based ECS implemented in PVSS. Light-weight software: Low CPU load to interfere as little as possible with event data crunching. Disk-less operation on the nodes. Low network traffic. Flexible service-like component: easy to change/add monitored variables by physicist-Gaudi-users (trigger people). Scalable with number of nodes Compatible with farm partitioning. ICALEPCS / Oct-03 Philippe Vannerem

GAUCHO: GAUdi Component Helping Online Gaucho package: monitor and control the Gaudi work on the farm. “users” (=future trigger algorithm programmers) can use this component in their Gaudi application to use the Gaudi-style Algorithm Service “MonitorSvc” and then declare the variables to be published in their algorithms. “MonitorSvc” publishes internal variables (integer, float, string) and histograms in the transient store using the DimMonitorServer. A DimPropertyServer gets/sets values of the properties of the ApplicationMgr and Algorithms (configuration, steering). A DimController with internal states receives commands via DIM and steers the ApplicationMgr. ICALEPCS / Oct-03 Philippe Vannerem

Gaudi-Gaucho-SCADA farm node - Gaudi DIM monitor node - PVSS DIM API ControlPanel ControlMgr MonitorPanel DIM API farm node - Gaudi HLT Algorithm ApplicationMgr HistogramSvc EventDataSvc DetectorDataSvc DimController MonitorSvc DimPropertySvr DimMonitorSvr Gaucho Optional: root GUI to display histograms ICALEPCS / Oct-03 Philippe Vannerem

Inter-process tool: DIM Distributed Information Management system Light-weight inter-process communication package based on the client/server paradigm Asynchronous interrupt-driven data transfer across platforms Here: C++ version used ICALEPCS / Oct-03 Philippe Vannerem

SCADA: controlling Gaudi apps PVSS control panel for EFF. ControlMgr runs control script Subfarm functionality: Distributed control of its nodes Farmnode functionality: Connect to DimController, which steers ApplicationMgr with commands (e.g. configure, start, pause,stop) Mapping of finite state machine in GaudiDimController in PVSS FSM control object Display tree of the configured sequence of Algorithms by talking to DimPropertySvr Get/Set Properties of Algorithms ICALEPCS / Oct-03 Philippe Vannerem

SCADA: monitoring Gaudi apps PVSS monitor panel for EFF. ControlMgr connects to DimH1D histos. Subfarm functionality: Add histograms from node together in one subfarm histogram. Farmnode functionality: PVSS control manager “pulls” each published value (float, int, string, histogram) at fixed time intervals. Panel displays monitored values Panel displays PVSS trend of monitored value ICALEPCS / Oct-03 Philippe Vannerem

Communicating histograms on-line GAUDI app SCADA app ROOT app Generalized concept of updated on-line histograms with DIM. Uses the AIDA interface classes for histograms (GAUDI-independent). Tested with ROOT histogram viewer. ROOT subscribes as a client to a DimH1D histogram, served by a Gaudi application, PVSS,… ICALEPCS / Oct-03 Philippe Vannerem

 philippe.vannerem@cern.ch Conclusions The LHCb design philosophy of a fully integrated hierarchical Experiment Control System is applied both to control hardware and software devices. GAUDI trigger algorithms running on the event filter farm will be controlled and monitored in a distributed way with a SCADA system. The interfacing package GAUCHO facilitates steering and monitoring of trigger algorithms for supervision and data quality checking purposes in an on-line environment.  philippe.vannerem@cern.ch ICALEPCS / Oct-03 Philippe Vannerem