1. Clara Gaspar, March 2006 LHCb’s Experiment Control System Step by Step

2. Clara Gaspar, March 2006 2 Overview ❚ LHCb’s Experiment Control System ❙ What do we (JCOP/LHCb) provide ❙ What sub-detectors/sub-systems need to implement ❚ PVSS & Framework reminder ❚ Interfacing Electronics Boards ❙ SPECS & CC-PC Tools ❙ The Configuration DB ❚ Hierarchical Control ❙ The FSM Toolkit ❚ Note: This tutorial is meant as an overview ❙ The PVSS & Framework and the FSM courses are still required in order to use the tools!

3. Clara Gaspar, March 2006 3 ECS Scope Detector Channels Front End Electronics Readout Network High Level Trigger Storage L0 Experiment Control System DAQ DCS Devices (HV, LV, GAS, Temperatures, etc.) External Systems (LHC, Technical Services, Safety, etc) TFC

4. Clara Gaspar, March 2006 4 ECS Generic Architecture... To Devices (HW or SW) Commands Status & Alarms ECS DCS DAQ DetDcs1 DetDcs N SubSys 1 SubSys 2 Dev 1 Dev 2 Dev 3 DetDaq 1 SubSysN Dev N LHC T.S.... GAS DSS Abstract levels

5. Clara Gaspar, March 2006 5 What do we provide? ❚ JCOP + LHCb Online provide: ❙ Not complete applications, but: ❙ A Framework, i.e. a set of tools to help sub- systems create their control systems: ❘ Complete, configurable components (ex. CAEN HV) ❘ Tools for defining User Components: 〡 Electronics boards (SPECS/ CC-PC) 〡 Specific equipment/software tasks (DIM protocol) ❘ Other Tools, for example: 〡 FSM for Building Hierarchies 〡 Configuration DB 〡 Archiving, Alarm handling, etc.

6. Clara Gaspar, March 2006 6 We also provide: ❚ Integration of Infrastructure Services: ❘ Power Distribution and Rack/Crate Control ❘ Cooling and Ventilation Control ❘ Magnet Control (Monitoring) ❘ Gas Control ❘ Detector Safety System ❚ And interface to: ❘ LHC machine ❘ Access Control System ❘ CERN Safety System ❚ Sub-detectors can use these components: ❙ For defining logic rules (using their states) ❙ For high-level operation (when applicable) ❘ Switch ON, Switch Off, Set parameters

7. Clara Gaspar, March 2006 7 And also Database Tools PVSS. To Offline... PVSS Cond.. DB Conf. DB To Offline... PVSS Arch. ❚ Interfaces to the three Logical Databases in the Online System Experimental Equipment

8. Clara Gaspar, March 2006 8 Online Database Contents ❙ Configuration DB contains: ❘ All data needed to configure the HW (or SW) for the various running modes 〡 Ex.: HV V0 Settings, Pedestal settings, trigger settings, etc. ❙ PVSS Archive contains: ❘ All monitoring data read from HW for monitoring and debugging of the Online System 〡 Ex.: HV Vmon Readings, temperatures, pedestal readings, etc. ❙ Conditions DB contains: ❘ A subset of the monitoring data read from HW if it is needed for Event processing (prob. packaged differently) 〡 Ex.: HV Vmon Readings if changed by more than n Volts ❘ Some configuration data once it has been used 〡 Ex.: Trigger settings used by a particular run

9. Clara Gaspar, March 2006 9 The Configuration DB ❚ The Configuration DB will contain: ❙ All "static" information about the devices ❘ Connectivity, addresses, etc. (also inventory and history) ➨ Developed within LHCb (supports queries) ❙ All "dynamic" data needed by the devices (for different running modes and different versions): ❘ Settings (voltages, alarm limits, etc.), Calibration constants, Pedestals, FPGA code (probably a pointer to it), etc. ➨ The settings for a particular running mode are called a “Recipe” (partial recipes available) ➨ The JCOP FW component implements a cache: 〡 Can be used without Oracle for tests 〡 Can pre-load several recipes before “Start of Run”

10. Clara Gaspar, March 2006 10 What needs to be done: ❚ Start bottom up ❙ Integrate each device into PVSS ❙ Define configuration recipes ❘ for the various running modes ❙ Build a hierarchy for each sub-system ❘ According to the guidelines ❙ Integrate the devices in the hierarchy

11. Clara Gaspar, March 2006 11 Device Integration ❚ Device Types ❙ HV & LV channels ❘ CAEN, ISEG, WIENNER -> JCOP Framework ❙ Analog inputs ❘ ELMB -> JCOP Framework ❙ Electronics boards ❘ SPECS & CC-PC -> Tools to describe boards ❘ TELL1 -> FW component (for common part) ❙ Other Components ❘ HW or SW -> FwDIM component ➨ Needs: PVSS, Framework, DIM,…

12. Clara Gaspar, March 2006 12 PVSS

13. Clara Gaspar, March 2006 13 PVSS Distribution

14. Clara Gaspar, March 2006 14 Datapoint Concept ❚ DP type -> DP Configs

15. Clara Gaspar, March 2006 15 Graphical Objects ❚ Reference Panels ❙ Can be “inherited” dynamically ❙ “$parameters” get replaced by instance value

16. Clara Gaspar, March 2006 16 Building User Interfaces ❚ Static Part -> Drag & Drop ❚ Dynamic part -> Control Scripts ("C" like) ❙ A few usefull calls for accessing DPs: ❘ dpGet (string dpName, value) ❘ dpSet (string dpName, value) ❘ dpConnect (string callback, string dpName) ❙ A few usefull calls for accessing Widgets: ❘ getValue (string widgetName, string widgetProperty, ) ❘ setValue (string widgetName, string widgetProperty, )

17. Clara Gaspar, March 2006 17 PVSS Features ❚ Open Architecture ❙ We can write our own managers ➨ It can be interfaced to anything (FSM, DIM) ❚ Highly Distributed ❙ 130 Systems (PCs) tested ➨ No major problem found ❚ Standard Interface ❙ All data of all sub-systems defined as DataPoints!

18. Clara Gaspar, March 2006 18 Demo-1 ❚ Start PVSS console ❚ Create a project (add installation tool) ❚ PVSS basic functionality ❙ PVSS Managers ❙ Parameterization Module ❘ Datapoint structures ❙ Graphic editor

19. Clara Gaspar, March 2006 19 Demo-2 ❚ Install Framework ❘ fwCore ❘ fwAnalogDigital ❘ fwCaen ❘ fwConfigurationDB ❘ fwDIM ❘ fwSpecs ❘ fwHw ❚ CAEN component: ❙ Create Crates/Boards/Channels ❘ “Crate0” will be used by FSM later ❙ Show Operation panels

20. Clara Gaspar, March 2006 20 DIM Distributed Information Management System ❙ Publish/Subscribe mechanism ❘ Servers publish Services. ❘ Clients subscribe to Services: 〡 On change or at regular intervals ❘ Clients can send commands to Servers ❙ Services ❘ A set of data 〡 any type or size 〡 Identified by a name ❙ A Name Server ❘ Keeps a list of available Services

21. Clara Gaspar, March 2006 21 DIM Some Characteristics ❙ Transparency ❘ DIM clients do not know where their interlocutors are. ❘ DIM components can move from one machine to another, all connections are transparently re-established. ❙ Available on mixed environments: ❘ UNIX (HP-UX, Sun-OS, Sun-Solaris, IBM-AIX, DEC-OSF, Linux), Windows, VMS, Real-time OSs (OS9, LynxOS, VxWorks) 〡 API available in “C”, C++ and Java ❙ Easy to Use ❘ One “call” and a process can become a server or a client. ❘ Monitoring and Visualization Tools Available. ❘ Documentation and examples at: http://www.cern.ch/dimhttp://www.cern.ch/dim

22. Clara Gaspar, March 2006 22 PVSS DIM ❚ FwDIM component: ❙ Server is a DIM Server ❙ Client is a PVSS Manager (PVSS00dim) ❙ Correspondence: PVSS DPs DIM Services ❘ Can be setup graphically via fwDIM panel ❘ Or via a script library ❙ When setup ❘ When Server updates Service data goes into DP ❘ Writing to DP will send a DIM Command ❙ Documentation at: ❘ http://www.cern.ch/lhcb-online/ecs/fw/FwDim.html http://www.cern.ch/lhcb-online/ecs/fw/FwDim.html

23. Clara Gaspar, March 2006 23 Non-standard components ❚ Integrating user components: ❙ Create a DIM server (C or C++) ❘ Publishes device status & data ❘ Receives Commands ❙ Create a PVSS Datapoint ❘ That matches the structure of DIM services ❙ Connect the DP to the DIM services ❘ Using the FwDIM tools ❙ Make a PVSS panel to control the device ❚ Used for: farm monitoring, trigger algorithms, etc.

24. Clara Gaspar, March 2006 24 Demo-3 ❚ FwDIM ❙ Configure DIM_DNS_NODE ❙ Start a DIM server (ex.: pvss_dim_server) ❙ Start DIM visualization tool ❘ DIMTree on Windows ❘ DID on Linux ❙ Start fwDIM.pnl ❘ Connect services to DPs ❘ Visualize from PVSS

25. Clara Gaspar, March 2006 25 Electronics Interface ❚ CC-PC & SPECS tools: ❙ Low-level Software ❘ A “C” library for accessing board components 〡 Via I2C, JTAG or parallel bus (and FPGA programming) ❘ A Generic DIM server for PVSS Communication ❙ PVSS Tools ( FW components: fwCcpc/fwSpecs ) ❘ A library (PVSS scripting) for accessing board components on any board with a CC-PC/Specs (equivalent to the low-level library) ❘ A graphical user interface providing the functionality available in the library

26. Clara Gaspar, March 2006 26 Electronics Integration ❚ Electronics Boards: ❙ Can use the CCPC/SPECS FW Tools for tests, but accessing the “chips” is not enough ❙ Boards have to be modeled in PVSS according to guidelines (ex. registers have to correspond to datapoints) in order to: ❘ Provide access to the Conf. DB 〡 Select a device/group of devices and say: Save as “Physics” recipe. ❘ Be able to archive the data ❘ Be able to send the data to the Cond. DB ❘ Integrate into the FSM, Generate alarms, etc.

27. Clara Gaspar, March 2006 27 Electronics Integration ❚ We provide a tool for modeling boards and their components (FWcomponent: FwHw) ❘ Declaring boards (access via SPECS or CC-PC) Containing: 〡 Groups of Chips (recursive) Containing: ❘ Chips (TTCrx, Beetle, etc.) Containing: ❘ Registers (access via I2C/JTAG/Parallel Bus) ❙ Contacts: ❘ Ricardo Fernandes: SPECS ❘ Stefan Koestner: CC-PC

28. Clara Gaspar, March 2006 28 Electronics boards ❚ Demo Setup SPECS Master SPECS Mezzanine I2C widget Croquette ServerPC: pclbcecs03 SPECS I2C ClientPC: Portable SpecsSrv PVSS PVSS00dim Ethernet DNS SupportPC: pclhcb155 PVSS PVSS00dim Note: The DNS should run on a stable machine (same as PVSS), not on a portable…

29. Clara Gaspar, March 2006 29 Demo-4 ❚ FwSpecs: ❙ Server PC: pclbcecs03 ❘ Configure DIM_DNS_NODE ❘ Start SpecsServer remotely ❙ Client PC: portable ❘ Configure DIM_DNS_NODE ❘ Start SpecsClient “direct access” panel 〡 Exercise I2C, JTAG, DCU ❘ Explain the “Monitoring” feature ❘ Show Advanced Panel (User Scripts) ❙ Documentation at (not this version yet): ❘ http://www.cern.ch/lhcb-online/ecs/PVSS_SPECS http://www.cern.ch/lhcb-online/ecs/PVSS_SPECS ❚ FwCcpc: very similar ❙ Tools will be presented at Online meeting

30. Clara Gaspar, March 2006 30 Custom Electronics ❚ Demo Example SPECS Master SPECS Mezzanine Velo Board Server PC SPECS SpecsSrv TTCrx Beetle1 Beetle2 I2C

31. Clara Gaspar, March 2006 31 Demo-5 ❚ FwHw ❙ Create HW types: ❘ TTCrx, Beetle and VeloBoard ❙ Configure Default Settings ❙ Create veloBoards ❘ “Operate” the board ❙ Interface to Configuration Database (cache) ❘ Save recipes (“PHYSICS”, “TEST”, etc.) ❘ Download recipes

32. Clara Gaspar, March 2006 32 Electronics guidelines ❚ FwHw: Some Guidelines ❙ If a “chip” has many registers ❘ If they can be written in one single operation 〡 Declare them as 1 register of size N 〡 This will optimize configuration time ❘ Some (a few) can also be declared separately 〡 If they are often accessed individually ❚ After using FwHw to define the boards: ❙ Design a user interface to operate each board type ❘ The library fwSpecs or fwCcpc will give you access to the data to be visualized or sent to the board ex.: fwSpecs_read(“board1.ttcrx1.reg2”,…)

33. Clara Gaspar, March 2006 33 Control Hierarchy... ECS DCS DAQ DetDcs1 DetDcs N SubSys 1 SubSys 2 Dev 1 Dev 2 Dev 3 DetDaq 1 SubSysN Dev N LHC T.S.... GAS DSS ❚ Building a Control Hierarchy ❙ And integrating Devices ❚ Needs: FwFSM, LHCb guidelines

34. Clara Gaspar, March 2006 34 Control Units ❚ Each node is able to: ❙ Summarize information (for the above levels) ❙ “Expand” actions (to the lower levels) ❙ Implement specific behaviour & Take local decisions ❘ Sequence & Automate operations ❘ Recover errors ❙ Include/Exclude children (i.e. partitioning) ❘ Excluded nodes can run is stand-alone ❙ User Interfacing ❘ Present information and receive commands DCS Tem p Tracke r Muon HVHV GA S HVHV

35. Clara Gaspar, March 2006 35 Device Units ❚ Device Units ❙ Provide the interface to real devices: ( Electronics Boards, HV channels, trigger algorithms, etc.) ❘ Can be enabled/disabled ❘ In order to integrate a device within FSM 〡 Deduce a STATE from device readings (in DPs) 〡 Implement COMMANDS as device settings ❘ Commands can apply the recipes previously defined Dev N

36. Clara Gaspar, March 2006 36 ❚ The FwFSM Component is based on: ❙ PVSS for: ❘ Device Description (Run-time Database) ❘ Device Access (OPC, Profibus, drivers) ❘ Alarm Handling (Generation, Filtering, Masking, etc) ❘ Archiving, Logging, Scripting, Trending ❘ User Interface Builder ❘ Alarm Display, Access Control, etc. ❙ SMI++ providing: ❘ Abstract behavior modeling (Finite State Machines) ❘ Automation & Error Recovery (Rule based system) The Control Framework Device Units Control Units

37. Clara Gaspar, March 2006 37 SMI++ ❚ Method ❙ Classes and Objects ❘ Allow the decomposition of a complex system into smaller manageable entities ❙ Finite State Machines ❘ Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS ❙ Rule-based reasoning ❘ Allow Automation and Error Recovery

38. Clara Gaspar, March 2006 38 SMI++ ❚ Method (Cont.) ❙ SMI++ Objects can be: ❘ Abstract (e.g. a Run or the DCS) ❘ Concrete (e.g. a power supply or a temp. sensor) ❙ Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts) ❙ Logically related objects can be grouped inside "SMI domains" representing a given sub-system (Framework: Control Unit)

39. Clara Gaspar, March 2006 39 SMI++ Run-time Environment Proxy Hardware Devices Obj SMI Domain Obj SMI Domain ❙ Device Level: Proxies ❘ drive the hardware: 〡 deduceState 〡 handleCommands ❘ C, C++, PVSS ctrl scripts ❙ Abstract Levels: Domains ❘ Implement the logical model ❘ Dedicated language - SML ❘ A C++ engine: smiSM ❙ User Interfaces ❘ For User Interaction ❙ All Tools available on: ❘ Windows, Unix (Linux) ❘ All communications are transparent and dynamically (re)established

40. Clara Gaspar, March 2006 40 SMI++ ❚ SMI++ - The Language ❙ SML –State Management Language ❘ Finite State Logic 〡 Objects are described as FSMs their main attribute is a STATE ❘ Parallelism 〡 Actions can be sent in parallel to several objects. Tests on the state of objects can block if the objects are still “transiting” ❘ Asynchronous Rules 〡 Actions can be triggered by logical conditions on the state of other objects

41. Clara Gaspar, March 2006 41 SML – The language ❚ Devices: ❚ Sub System: ❚ Objects can be dynamically included/excluded in a Set

42. Clara Gaspar, March 2006 42 SML example (automation) ❚ External Device: ❚ Sub System:

43. Clara Gaspar, March 2006 43 PVSS/SMI++ Integration ❚ Graphical Configuration of SMI++ Using PVSS

44. Clara Gaspar, March 2006 44 Building Hierarchies ❚ Hierarchy of CUs ❙ Distributed over several machines ❘ "&" means reference to a CU in another system ❙ Editor Mode: ❘ Add / Remove / Change Settings ❙ Navigator Mode ❘ Start / Stop / View

45. Clara Gaspar, March 2006 45 Control Unit Run-Time ❚ Dynamically generated operation panels (Uniform look and feel) ❚ Configurable User Panels

46. Clara Gaspar, March 2006 46 Features of PVSS/SMI++ ❚ Task Separation: ❙ SMI Proxies/PVSS Scripts execute only basic actions – No intelligence ❙ SMI Objects implement the logic behaviour ❙ Advantages: ❘ Change the HW -> change only PVSS ❘ Change logic behaviour sequencing and dependency of actions, etc -> change only SMI rules

47. Clara Gaspar, March 2006 47 Features of PVSS/SMI++ ❚ Error Recovery Mechanism ❙ Bottom Up ❘ SMI Objects react to changes of their children 〡 In an event-driven, asynchronous, fashion ❙ Distributed ❘ Each Sub-System recovers its errors 〡 Each team knows how to recover local errors ❙ Hierarchical/Parallel recovery ❙ Can provide complete automation even for very large systems

48. Clara Gaspar, March 2006 48 Demo-6 ❚ Show a simple Hierarchy ❙ Install fwLHCb_FsmDomains ❙ In installs standard LHCb FSM Domain Types ❙ And it creates: ❙ Show Include/Exclude and Enable/Disable ❙ Show “Temp” FSM and Alarm Handling VELO DCS VELO Motors VELO Temp

49. Clara Gaspar, March 2006 49 Sub-detector FSM Guidelines ❚ Started defining naming conventions. ❚ Defined standard “domains” per sub-detector: ❙ DCS ❘ DCS Infrastructure (Cooling, Gas, Temperatures, pressures, etc) that is normally stable throughout a running period ❙ HV ❘ High Voltages or in general components that depend on the status of the LHC machine (fill related) ❙ DAQ ❘ All Electronics and components necessary to take data (run related) ❙ DAQI ❘ Infrastructure necessary for the DAQ to work (computers, networks, electrical power, etc.) in general also stable throughout a running period. ❚ And standard states & transitions per domain. ❚ Doc available in EDMS: ❘ https://edms.cern.ch/document/655828/1 https://edms.cern.ch/document/655828/1

50. Clara Gaspar, March 2006 50 FSM Guidelines ❚ State Diagram for Trigger and DAQ Domains: ❙ Possible intermediate “CONFIGURING” and “STARTING” states if operations slow…

51. Clara Gaspar, March 2006 51 MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECS VELO Dev1 VELO DevN SubFarm 1 SubFarm N

52. Clara Gaspar, March 2006 52 MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy & Conf. DB VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECS VELO Dev1 VELO DevN Conf. DB 3 2 1 1 Configure/mode=“PHYSICS” (Get “PHYSICS” Settings) Apply Settings 2 3 1 1

53. Clara Gaspar, March 2006 53 Demo-7 ❚ Using type: DAQ_Domain ❚ Create: ❚ Create type VeloBoard ❚ Integrate veloBoard1, veloBoard2,… ❙ Apply recipes on “Configure” command ❙ Note: There will be a “configurator” object per CU which gets recipes from DB to cache VELO DAQ VELO Dev1 VELO Board1 VELO FEE VELO TELL1

54. Clara Gaspar, March 2006 54 Demo-8 ❚ Using type: ECS_Domain ❚ Create: VELO DAQ VELO Dev1 VELO Board1 VELO FEE VELO TELL1 VELO DCS VELO Motors VELO Temp VELO HV VELO

55. Clara Gaspar, March 2006 55 The End ❚ Questions?

56. Clara Gaspar, March 2006 56 MUON DCS MUON HV MUON DAQI MUON DAQ Hierarchy & Partitioning VELO DCS Infrast.DCSHVDAQIDAQL0TFCHLTLHC VELO HV VELO DAQI VELO DAQ VELO DCS_1 VELO DCS_2 VELO DAQ_1 VELO DAQ_2 ECSVELO