1. Philippe Vannerem CERN / EP ICALEPCS - Oct03
Distributed Control and Monitoring of High-Level Trigger Processes on the LHCb on-line Farm
Philippe Vannerem
CERN / EP
ICALEPCS - Oct03

2. Contents the Experiment Control System of LHCb the Event Filter Farm
GAUDI, the software framework for event processing
GAUCHO, how to glue GAUDI to SCADA
Communicating histograms
Conclusions
ICALEPCS / Oct-03
Philippe Vannerem

3. ECS Scope DCS Devices (HV, LV, GAS, Temperatures, etc.)
Detector Channels
Trigger
Front End Electronics
Experiment Control System
Readout Network
Processing/Filtering
Storage
DAQ
External Systems (LHC, Technical Services, Safety, etc)

4. ECS: Generic SW Architecture
T.S.
LHC
DCS
DAQ
DSS
...
...
GAS
DetDcs1
DetDcsN
DetDaq1
Status & Alarms
Commands
SubSys1
SubSys2
SubSysN
Dev1
Dev2
Dev3
DevN
To Devices (Hardware or Software)
Distributed Hierarchical Control
Commercial components and industrial standards
ICALEPCS / Oct-03
Philippe Vannerem

5. ECS: Hierarchical control
Each node is able to control and monitor its subtrees:
Propagate commands to child level
Summarize information for the parent level
Implement specific behaviour & Take local decisions
Sequence & Automate operations
Recover errors
Partitioning
Exclude one or more of its subtrees
which then can run stand-alone
User Interfacing
Present information and receive commands
Filter and display alarms
DCS
Tracker
Muon HV
Temp HV
GAS
ICALEPCS / Oct-03
Philippe Vannerem

6. ECS: implementation
Implementation in commercial SCADA system software: PVSS II
device description + access (OPC, Profibus, drivers)
alarm handling+display
user interface builder + control scripting
archiving
Joint COntrols Project between the LHC experiments:
The Framework: hierarchical control templates + tools and control modules for common HW in PVSS
Important building block: SMI++:
State Management Interface, distributed control
Abstract behaviour modeling Finite State Machines
Automation & Error recovery with a Rule based system
LHCb ECS is based the JCOP Framework and will extend it for its specific needs.
ICALEPCS / Oct-03
Philippe Vannerem

7. LHCb’s DAQ&trigger architecture
Level-1 Traffic
HLT Traffic
Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE
TRM
126 Links
1100 / 25 = 44 kHz
5.5 GB/s
323 Links
40 / 10 = 4 kHz
1.6 GB/s
Multiplexing Layer
Switch
Switch
Switch
Switch
Switch
64 Links
32 Links
Readout Network
L1-Decision
Sorter
TFC System
94 Links
7.1 GB/s
Storage System … 94
SFCs
SFC
Switch
CPU
SFC
Switch
CPU
SFC
Switch
CPU
SFC
Switch
CPU
CPU Farm
Gb Ethernet
Level-1 Traffic
Mixed Traffic
HLT Traffic
~1800 CPUs

8. Event Filter Farm Control
EFF: ~1800 commodity rack-mounted CPUs
CPUs+SFCs linked to ECS PCs in separate Ethernet LAN
ECS Tasks:
Hardware control (remote boot, reset,…) + monitoring (CPU usage, memory, IO, etc.)
Software control + monitoring
Level-1+ High-Level Trigger algorithms are implemented using the GAUDI framework
Event Builder Switch
SFC
CPU .
. . .
ICALEPCS / Oct-03
Philippe Vannerem

9. GAUDI Framework: design choices
Separation between data and algorithms
Separation between “transient” and “persistent” data
Generic component interfaces
OO Framework implemented with C++, runs on Win2000+Linux
ICALEPCS / Oct-03
Philippe Vannerem

10. GAUDI: component architecture
Interface defined in C++ pure virtual class
User specifies the components to be used in the Joboptions file
DLLs are loaded at run-time
Algorithms can be configured in the joboptions file
ICALEPCS / Oct-03
Philippe Vannerem

11. Interface Requirements
Real-time system to use on the EFF
Interface to the trigger algorithms written in Gaudi event data processing framework.
Interface to the SCADA-based ECS implemented in PVSS.
Light-weight software:
Low CPU load to interfere as little as possible with event data crunching.
Disk-less operation on the nodes.
Low network traffic.
Flexible service-like component: easy to change/add monitored variables by physicist-Gaudi-users (trigger people).
Scalable with number of nodes
Compatible with farm partitioning.
ICALEPCS / Oct-03
Philippe Vannerem

12. GAUCHO: GAUdi Component Helping Online
Gaucho package: monitor and control the Gaudi work on the farm.
“users” (=future trigger algorithm programmers) can use this component in their Gaudi application to use the Gaudi-style Algorithm Service “MonitorSvc” and then declare the variables to be published in their algorithms.
“MonitorSvc” publishes internal variables (integer, float, string) and histograms in the transient store using the DimMonitorServer.
A DimPropertyServer gets/sets values of the properties of the ApplicationMgr and Algorithms (configuration, steering).
A DimController with internal states receives commands via DIM and steers the ApplicationMgr.
ICALEPCS / Oct-03
Philippe Vannerem

13. Gaudi-Gaucho-SCADA farm node - Gaudi DIM monitor node - PVSS DIM API
ControlPanel
ControlMgr
MonitorPanel
DIM API
farm node - Gaudi
HLT Algorithm
ApplicationMgr
HistogramSvc
EventDataSvc
DetectorDataSvc
DimController
MonitorSvc
DimPropertySvr
DimMonitorSvr
Gaucho
Optional: root GUI to display histograms
ICALEPCS / Oct-03
Philippe Vannerem

14. Inter-process tool: DIM
Distributed Information Management system
Light-weight inter-process communication package based on the client/server paradigm
Asynchronous interrupt-driven data transfer across platforms
Here: C++ version used
ICALEPCS / Oct-03
Philippe Vannerem

15. SCADA: controlling Gaudi apps
PVSS control panel for EFF.
ControlMgr runs control script
Subfarm functionality:
Distributed control of its nodes
Farmnode functionality:
Connect to DimController, which steers ApplicationMgr with commands (e.g. configure, start, pause,stop)
Mapping of finite state machine in GaudiDimController in PVSS FSM control object
Display tree of the configured sequence of Algorithms by talking to DimPropertySvr
Get/Set Properties of Algorithms
ICALEPCS / Oct-03
Philippe Vannerem

16. SCADA: monitoring Gaudi apps
PVSS monitor panel for EFF.
ControlMgr connects to DimH1D histos.
Subfarm functionality:
Add histograms from node together in one subfarm histogram.
Farmnode functionality:
PVSS control manager “pulls” each published value (float, int, string, histogram) at fixed time intervals.
Panel displays monitored values
Panel displays PVSS trend of monitored value
ICALEPCS / Oct-03
Philippe Vannerem

17. Communicating histograms on-line
GAUDI app
SCADA app
ROOT app
Generalized concept of updated on-line histograms with DIM.
Uses the AIDA interface classes for histograms (GAUDI-independent).
Tested with ROOT histogram viewer.
ROOT subscribes as a client to a DimH1D histogram, served by a Gaudi application, PVSS,…
ICALEPCS / Oct-03
Philippe Vannerem

18.  philippe.vannerem@cern.ch
Conclusions
The LHCb design philosophy of a fully integrated hierarchical Experiment Control System is applied both to control hardware and software devices.
GAUDI trigger algorithms running on the event filter farm will be controlled and monitored in a distributed way with a SCADA system.
The interfacing package GAUCHO facilitates steering and monitoring of trigger algorithms for supervision and data quality checking purposes in an on-line environment. 
ICALEPCS / Oct-03
Philippe Vannerem