Clara Gaspar, March 2006 LHCb’s Experiment Control System Step by Step

Clara Gaspar, March Overview ❚ LHCb’s Experiment Control System ❙ What do we (JCOP/LHCb) provide ❙ What sub-detectors/sub-systems need to implement ❚ PVSS & Framework reminder ❚ Interfacing Electronics Boards ❙ SPECS & CC-PC Tools ❙ The Configuration DB ❚ Hierarchical Control ❙ The FSM Toolkit ❚ Note: This tutorial is meant as an overview ❙ The PVSS & Framework and the FSM courses are still required in order to use the tools!

Clara Gaspar, March ECS Scope Detector Channels Front End Electronics Readout Network High Level Trigger Storage L0 Experiment Control System DAQ DCS Devices (HV, LV, GAS, Temperatures, etc.) External Systems (LHC, Technical Services, Safety, etc) TFC

Clara Gaspar, March ECS Generic Architecture... To Devices (HW or SW) Commands Status & Alarms ECS DCS DAQ DetDcs1 DetDcs N SubSys 1 SubSys 2 Dev 1 Dev 2 Dev 3 DetDaq 1 SubSysN Dev N LHC T.S.... GAS DSS Abstract levels

Clara Gaspar, March What do we provide? ❚ JCOP + LHCb Online provide: ❙ Not complete applications, but: ❙ A Framework, i.e. a set of tools to help sub- systems create their control systems: ❘ Complete, configurable components (ex. CAEN HV) ❘ Tools for defining User Components: 〡 Electronics boards (SPECS/ CC-PC) 〡 Specific equipment/software tasks (DIM protocol) ❘ Other Tools, for example: 〡 FSM for Building Hierarchies 〡 Configuration DB 〡 Archiving, Alarm handling, etc.

Clara Gaspar, March We also provide: ❚ Integration of Infrastructure Services: ❘ Power Distribution and Rack/Crate Control ❘ Cooling and Ventilation Control ❘ Magnet Control (Monitoring) ❘ Gas Control ❘ Detector Safety System ❚ And interface to: ❘ LHC machine ❘ Access Control System ❘ CERN Safety System ❚ Sub-detectors can use these components: ❙ For defining logic rules (using their states) ❙ For high-level operation (when applicable) ❘ Switch ON, Switch Off, Set parameters

Clara Gaspar, March And also Database Tools PVSS. To Offline... PVSS Cond.. DB Conf. DB To Offline... PVSS Arch. ❚ Interfaces to the three Logical Databases in the Online System Experimental Equipment

Clara Gaspar, March Online Database Contents ❙ Configuration DB contains: ❘ All data needed to configure the HW (or SW) for the various running modes 〡 Ex.: HV V0 Settings, Pedestal settings, trigger settings, etc. ❙ PVSS Archive contains: ❘ All monitoring data read from HW for monitoring and debugging of the Online System 〡 Ex.: HV Vmon Readings, temperatures, pedestal readings, etc. ❙ Conditions DB contains: ❘ A subset of the monitoring data read from HW if it is needed for Event processing (prob. packaged differently) 〡 Ex.: HV Vmon Readings if changed by more than n Volts ❘ Some configuration data once it has been used 〡 Ex.: Trigger settings used by a particular run

Clara Gaspar, March The Configuration DB ❚ The Configuration DB will contain: ❙ All "static" information about the devices ❘ Connectivity, addresses, etc. (also inventory and history) ➨ Developed within LHCb (supports queries) ❙ All "dynamic" data needed by the devices (for different running modes and different versions): ❘ Settings (voltages, alarm limits, etc.), Calibration constants, Pedestals, FPGA code (probably a pointer to it), etc. ➨ The settings for a particular running mode are called a “Recipe” (partial recipes available) ➨ The JCOP FW component implements a cache: 〡 Can be used without Oracle for tests 〡 Can pre-load several recipes before “Start of Run”

Clara Gaspar, March What needs to be done: ❚ Start bottom up ❙ Integrate each device into PVSS ❙ Define configuration recipes ❘ for the various running modes ❙ Build a hierarchy for each sub-system ❘ According to the guidelines ❙ Integrate the devices in the hierarchy

Clara Gaspar, March Device Integration ❚ Device Types ❙ HV & LV channels ❘ CAEN, ISEG, WIENNER -> JCOP Framework ❙ Analog inputs ❘ ELMB -> JCOP Framework ❙ Electronics boards ❘ SPECS & CC-PC -> Tools to describe boards ❘ TELL1 -> FW component (for common part) ❙ Other Components ❘ HW or SW -> FwDIM component ➨ Needs: PVSS, Framework, DIM,…

Clara Gaspar, March PVSS

Clara Gaspar, March PVSS Distribution

Clara Gaspar, March Datapoint Concept ❚ DP type -> DP Configs

Clara Gaspar, March Graphical Objects ❚ Reference Panels ❙ Can be “inherited” dynamically ❙ “$parameters” get replaced by instance value

Clara Gaspar, March Building User Interfaces ❚ Static Part -> Drag & Drop ❚ Dynamic part -> Control Scripts ("C" like) ❙ A few usefull calls for accessing DPs: ❘ dpGet (string dpName, value) ❘ dpSet (string dpName, value) ❘ dpConnect (string callback, string dpName) ❙ A few usefull calls for accessing Widgets: ❘ getValue (string widgetName, string widgetProperty, ) ❘ setValue (string widgetName, string widgetProperty, )

Clara Gaspar, March PVSS Features ❚ Open Architecture ❙ We can write our own managers ➨ It can be interfaced to anything (FSM, DIM) ❚ Highly Distributed ❙ 130 Systems (PCs) tested ➨ No major problem found ❚ Standard Interface ❙ All data of all sub-systems defined as DataPoints!

Clara Gaspar, March Demo-1 ❚ Start PVSS console ❚ Create a project (add installation tool) ❚ PVSS basic functionality ❙ PVSS Managers ❙ Parameterization Module ❘ Datapoint structures ❙ Graphic editor

Clara Gaspar, March Demo-2 ❚ Install Framework ❘ fwCore ❘ fwAnalogDigital ❘ fwCaen ❘ fwConfigurationDB ❘ fwDIM ❘ fwSpecs ❘ fwHw ❚ CAEN component: ❙ Create Crates/Boards/Channels ❘ “Crate0” will be used by FSM later ❙ Show Operation panels

Clara Gaspar, March DIM Distributed Information Management System ❙ Publish/Subscribe mechanism ❘ Servers publish Services. ❘ Clients subscribe to Services: 〡 On change or at regular intervals ❘ Clients can send commands to Servers ❙ Services ❘ A set of data 〡 any type or size 〡 Identified by a name ❙ A Name Server ❘ Keeps a list of available Services

Clara Gaspar, March DIM Some Characteristics ❙ Transparency ❘ DIM clients do not know where their interlocutors are. ❘ DIM components can move from one machine to another, all connections are transparently re-established. ❙ Available on mixed environments: ❘ UNIX (HP-UX, Sun-OS, Sun-Solaris, IBM-AIX, DEC-OSF, Linux), Windows, VMS, Real-time OSs (OS9, LynxOS, VxWorks) 〡 API available in “C”, C++ and Java ❙ Easy to Use ❘ One “call” and a process can become a server or a client. ❘ Monitoring and Visualization Tools Available. ❘ Documentation and examples at:

Clara Gaspar, March PVSS DIM ❚ FwDIM component: ❙ Server is a DIM Server ❙ Client is a PVSS Manager (PVSS00dim) ❙ Correspondence: PVSS DPs DIM Services ❘ Can be setup graphically via fwDIM panel ❘ Or via a script library ❙ When setup ❘ When Server updates Service data goes into DP ❘ Writing to DP will send a DIM Command ❙ Documentation at: ❘

Clara Gaspar, March Non-standard components ❚ Integrating user components: ❙ Create a DIM server (C or C++) ❘ Publishes device status & data ❘ Receives Commands ❙ Create a PVSS Datapoint ❘ That matches the structure of DIM services ❙ Connect the DP to the DIM services ❘ Using the FwDIM tools ❙ Make a PVSS panel to control the device ❚ Used for: farm monitoring, trigger algorithms, etc.

Clara Gaspar, March Demo-3 ❚ FwDIM ❙ Configure DIM_DNS_NODE ❙ Start a DIM server (ex.: pvss_dim_server) ❙ Start DIM visualization tool ❘ DIMTree on Windows ❘ DID on Linux ❙ Start fwDIM.pnl ❘ Connect services to DPs ❘ Visualize from PVSS

Clara Gaspar, March Electronics Interface ❚ CC-PC & SPECS tools: ❙ Low-level Software ❘ A “C” library for accessing board components 〡 Via I2C, JTAG or parallel bus (and FPGA programming) ❘ A Generic DIM server for PVSS Communication ❙ PVSS Tools ( FW components: fwCcpc/fwSpecs ) ❘ A library (PVSS scripting) for accessing board components on any board with a CC-PC/Specs (equivalent to the low-level library) ❘ A graphical user interface providing the functionality available in the library

Clara Gaspar, March Electronics Integration ❚ Electronics Boards: ❙ Can use the CCPC/SPECS FW Tools for tests, but accessing the “chips” is not enough ❙ Boards have to be modeled in PVSS according to guidelines (ex. registers have to correspond to datapoints) in order to: ❘ Provide access to the Conf. DB 〡 Select a device/group of devices and say: Save as “Physics” recipe. ❘ Be able to archive the data ❘ Be able to send the data to the Cond. DB ❘ Integrate into the FSM, Generate alarms, etc.

Clara Gaspar, March Electronics Integration ❚ We provide a tool for modeling boards and their components (FWcomponent: FwHw) ❘ Declaring boards (access via SPECS or CC-PC) Containing: 〡 Groups of Chips (recursive) Containing: ❘ Chips (TTCrx, Beetle, etc.) Containing: ❘ Registers (access via I2C/JTAG/Parallel Bus) ❙ Contacts: ❘ Ricardo Fernandes: SPECS ❘ Stefan Koestner: CC-PC

Clara Gaspar, March Electronics boards ❚ Demo Setup SPECS Master SPECS Mezzanine I2C widget Croquette ServerPC: pclbcecs03 SPECS I2C ClientPC: Portable SpecsSrv PVSS PVSS00dim Ethernet DNS SupportPC: pclhcb155 PVSS PVSS00dim Note: The DNS should run on a stable machine (same as PVSS), not on a portable…

Clara Gaspar, March Demo-4 ❚ FwSpecs: ❙ Server PC: pclbcecs03 ❘ Configure DIM_DNS_NODE ❘ Start SpecsServer remotely ❙ Client PC: portable ❘ Configure DIM_DNS_NODE ❘ Start SpecsClient “direct access” panel 〡 Exercise I2C, JTAG, DCU ❘ Explain the “Monitoring” feature ❘ Show Advanced Panel (User Scripts) ❙ Documentation at (not this version yet): ❘ ❚ FwCcpc: very similar ❙ Tools will be presented at Online meeting

Clara Gaspar, March Custom Electronics ❚ Demo Example SPECS Master SPECS Mezzanine Velo Board Server PC SPECS SpecsSrv TTCrx Beetle1 Beetle2 I2C

Clara Gaspar, March Demo-5 ❚ FwHw ❙ Create HW types: ❘ TTCrx, Beetle and VeloBoard ❙ Configure Default Settings ❙ Create veloBoards ❘ “Operate” the board ❙ Interface to Configuration Database (cache) ❘ Save recipes (“PHYSICS”, “TEST”, etc.) ❘ Download recipes

Clara Gaspar, March Electronics guidelines ❚ FwHw: Some Guidelines ❙ If a “chip” has many registers ❘ If they can be written in one single operation 〡 Declare them as 1 register of size N 〡 This will optimize configuration time ❘ Some (a few) can also be declared separately 〡 If they are often accessed individually ❚ After using FwHw to define the boards: ❙ Design a user interface to operate each board type ❘ The library fwSpecs or fwCcpc will give you access to the data to be visualized or sent to the board ex.: fwSpecs_read(“board1.ttcrx1.reg2”,…)

Clara Gaspar, March Control Hierarchy... ECS DCS DAQ DetDcs1 DetDcs N SubSys 1 SubSys 2 Dev 1 Dev 2 Dev 3 DetDaq 1 SubSysN Dev N LHC T.S.... GAS DSS ❚ Building a Control Hierarchy ❙ And integrating Devices ❚ Needs: FwFSM, LHCb guidelines

Clara Gaspar, March Control Units ❚ Each node is able to: ❙ Summarize information (for the above levels) ❙ “Expand” actions (to the lower levels) ❙ Implement specific behaviour & Take local decisions ❘ Sequence & Automate operations ❘ Recover errors ❙ Include/Exclude children (i.e. partitioning) ❘ Excluded nodes can run is stand-alone ❙ User Interfacing ❘ Present information and receive commands DCS Tem p Tracke r Muon HVHV GA S HVHV

Clara Gaspar, March Device Units ❚ Device Units ❙ Provide the interface to real devices: ( Electronics Boards, HV channels, trigger algorithms, etc.) ❘ Can be enabled/disabled ❘ In order to integrate a device within FSM 〡 Deduce a STATE from device readings (in DPs) 〡 Implement COMMANDS as device settings ❘ Commands can apply the recipes previously defined Dev N

Clara Gaspar, March ❚ The FwFSM Component is based on: ❙ PVSS for: ❘ Device Description (Run-time Database) ❘ Device Access (OPC, Profibus, drivers) ❘ Alarm Handling (Generation, Filtering, Masking, etc) ❘ Archiving, Logging, Scripting, Trending ❘ User Interface Builder ❘ Alarm Display, Access Control, etc. ❙ SMI++ providing: ❘ Abstract behavior modeling (Finite State Machines) ❘ Automation & Error Recovery (Rule based system) The Control Framework Device Units Control Units

Clara Gaspar, March SMI++ ❚ Method ❙ Classes and Objects ❘ Allow the decomposition of a complex system into smaller manageable entities ❙ Finite State Machines ❘ Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS ❙ Rule-based reasoning ❘ Allow Automation and Error Recovery

Clara Gaspar, March SMI++ ❚ Method (Cont.) ❙ SMI++ Objects can be: ❘ Abstract (e.g. a Run or the DCS) ❘ Concrete (e.g. a power supply or a temp. sensor) ❙ Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts) ❙ Logically related objects can be grouped inside "SMI domains" representing a given sub-system (Framework: Control Unit)

Clara Gaspar, March SMI++ Run-time Environment Proxy Hardware Devices Obj SMI Domain Obj SMI Domain ❙ Device Level: Proxies ❘ drive the hardware: 〡 deduceState 〡 handleCommands ❘ C, C++, PVSS ctrl scripts ❙ Abstract Levels: Domains ❘ Implement the logical model ❘ Dedicated language - SML ❘ A C++ engine: smiSM ❙ User Interfaces ❘ For User Interaction ❙ All Tools available on: ❘ Windows, Unix (Linux) ❘ All communications are transparent and dynamically (re)established

Clara Gaspar, March SMI++ ❚ SMI++ - The Language ❙ SML –State Management Language ❘ Finite State Logic 〡 Objects are described as FSMs their main attribute is a STATE ❘ Parallelism 〡 Actions can be sent in parallel to several objects. Tests on the state of objects can block if the objects are still “transiting” ❘ Asynchronous Rules 〡 Actions can be triggered by logical conditions on the state of other objects

Clara Gaspar, March SML – The language ❚ Devices: ❚ Sub System: ❚ Objects can be dynamically included/excluded in a Set

Clara Gaspar, March SML example (automation) ❚ External Device: ❚ Sub System:

Clara Gaspar, March PVSS/SMI++ Integration ❚ Graphical Configuration of SMI++ Using PVSS

Clara Gaspar, March Building Hierarchies ❚ Hierarchy of CUs ❙ Distributed over several machines ❘ "&" means reference to a CU in another system ❙ Editor Mode: ❘ Add / Remove / Change Settings ❙ Navigator Mode ❘ Start / Stop / View

Clara Gaspar, March Control Unit Run-Time ❚ Dynamically generated operation panels (Uniform look and feel) ❚ Configurable User Panels

Clara Gaspar, March Features of PVSS/SMI++ ❚ Task Separation: ❙ SMI Proxies/PVSS Scripts execute only basic actions – No intelligence ❙ SMI Objects implement the logic behaviour ❙ Advantages: ❘ Change the HW -> change only PVSS ❘ Change logic behaviour sequencing and dependency of actions, etc -> change only SMI rules

Clara Gaspar, March Features of PVSS/SMI++ ❚ Error Recovery Mechanism ❙ Bottom Up ❘ SMI Objects react to changes of their children 〡 In an event-driven, asynchronous, fashion ❙ Distributed ❘ Each Sub-System recovers its errors 〡 Each team knows how to recover local errors ❙ Hierarchical/Parallel recovery ❙ Can provide complete automation even for very large systems

Clara Gaspar, March Demo-6 ❚ Show a simple Hierarchy ❙ Install fwLHCb_FsmDomains ❙ In installs standard LHCb FSM Domain Types ❙ And it creates: ❙ Show Include/Exclude and Enable/Disable ❙ Show “Temp” FSM and Alarm Handling VELO DCS VELO Motors VELO Temp

Clara Gaspar, March Sub-detector FSM Guidelines ❚ Started defining naming conventions. ❚ Defined standard “domains” per sub-detector: ❙ DCS ❘ DCS Infrastructure (Cooling, Gas, Temperatures, pressures, etc) that is normally stable throughout a running period ❙ HV ❘ High Voltages or in general components that depend on the status of the LHC machine (fill related) ❙ DAQ ❘ All Electronics and components necessary to take data (run related) ❙ DAQI ❘ Infrastructure necessary for the DAQ to work (computers, networks, electrical power, etc.) in general also stable throughout a running period. ❚ And standard states & transitions per domain. ❚ Doc available in EDMS: ❘

Clara Gaspar, March FSM Guidelines ❚ State Diagram for Trigger and DAQ Domains: ❙ Possible intermediate “CONFIGURING” and “STARTING” states if operations slow…

Clara Gaspar, March MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECS VELO Dev1 VELO DevN SubFarm 1 SubFarm N

Clara Gaspar, March MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy & Conf. DB VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECS VELO Dev1 VELO DevN Conf. DB Configure/mode=“PHYSICS” (Get “PHYSICS” Settings) Apply Settings

Clara Gaspar, March Demo-7 ❚ Using type: DAQ_Domain ❚ Create: ❚ Create type VeloBoard ❚ Integrate veloBoard1, veloBoard2,… ❙ Apply recipes on “Configure” command ❙ Note: There will be a “configurator” object per CU which gets recipes from DB to cache VELO DAQ VELO Dev1 VELO Board1 VELO FEE VELO TELL1

Clara Gaspar, March Demo-8 ❚ Using type: ECS_Domain ❚ Create: VELO DAQ VELO Dev1 VELO Board1 VELO FEE VELO TELL1 VELO DCS VELO Motors VELO Temp VELO HV VELO

Clara Gaspar, March The End ❚ Questions?

Clara Gaspar, March MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy & Partitioning VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECSVELO