1. The Online Detector Control at the BaBar experiment at SLAC
Pierluigi Paolucci
I.N.F.N. of Napoli
Introduction;
DCS architecture;
EPICS;
Component Proxy;
Database;
Conclusions.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

2. Pigi Paolucci - I.N.F.N. of Napoli
BaBar detector
BaBar consists of 5
sub-detectors:
Silicon Vertex Tracker;
Drift Chamber;
Detector of Internally Reflected Cherenkov ;
ElectroMagnetic Calorimeter;
Instrumented Flux Return.
m detection and neutral hadron identification
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

3. BaBar DCS architecture
Online 
EPICS 
Hardware 
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

4. Pigi Paolucci - I.N.F.N. of Napoli
DCS requirements
Responsibilities of the detector control system:
Safe and reliable operation of BaBar detector;
Safe and reliable operation of PEP II accelerator;
Monitor BaBar and PEP II operations;
Generate Alarms and Log files.
Report status of detector to the online system and PEP II;
Configure the detector for calibration/physics data taking;
Archive ambient data.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

5. Architecture and standard hardware
We have adopted some hardware standards to focus operational expertise, to limit device-dependent code development, and to ease maintenance.
The standard DCS crate is a 6U VME chassis, it is equipped with a Motorola MVME177 single board running VxWorks as operating system.
Analog data are digitized by VME boards or directly on the detector. In this latter case, CANbus is the standard bus solution.
A Greensprings carrier board receive and transmit the CAN messages.
A general monitor board has been developed to interface the CANbus to the electronic placed on the detector.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

6. Pigi Paolucci - I.N.F.N. of Napoli
Slave Crate
Fast-OR signals
Detector
HV system
Analog signals
CaenNET
CAN-bus
Electronic house
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

7. Experimental Physics and Industrial Control System
EPICS is used to control and monitoring the detector subsystems, environmental conditions, and accelerator parameters.
EPICS is a freely distributed software package that includes a real-time kernel running under the VxWorks operating system, drivers for commonly used hardware components (HV, I/O, TDC, ADC, CANbus…), and a suite of user applications for control, monitoring, and data analysis.
EPICS source code is written in C language and is available for customization by the user.
EPICS architecture is distributed, with the Input/Output Controller (IOC) as the fundamental component.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

8. Pigi Paolucci - I.N.F.N. of Napoli
DCS Architecture
The BaBar DCS is a distributed architecture based on a TCP/IP network of 15 IOCs containing about 105 hardware and software channels describing the status of each subsystem and central system (gas and cooling system, electronic house, interaction hall….)
Each detector has one or more IOCs (VME + CPU) running is own EPCIS software code and is able to communicate with all the others IOCs.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

9. Pigi Paolucci - I.N.F.N. of Napoli
EPICS, Channel Access
EPICS toll-kit contains a lot of tools connected via the Channel Access client/server libraries.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

10. Channel Access performance
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

11. Pigi Paolucci - I.N.F.N. of Napoli
EPICS framework
EDD/DM
Database
EPICS
Control layer
Alarm Handler
Sequencer
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

12. Pigi Paolucci - I.N.F.N. of Napoli
EPICS software tools I
The EPICS internal database is used to define and store the records (process block) records use around 1 Mbytes.
For each records it is possible to define a lot of fields and the most useful are:
scan period (max 60Hz), alarm limit and severity Hardware
units, precision, conversion, link Software
Alarm, state machine and monitor EPICS tools
Muon system has 2600 hardware records and 1400 software records used to:
Monitor HV, LV, Gas and Cooling systems, environmental parameters, noise and front-end boards.
summarize the results and the status of the IFR and of each partition (sector).
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

13. EPICS software tools II
The State Notation Language (similar to the C language) is used to build state machines.
The I/O variables of the state machines have to be defined as EPICS records and they can be load from different IOCs. The state machines run in the VxWorks environment.
For the visualization has been used the EDD/DM tool that provides an interactive display editor and high performance display manager; more different tools are available from JAVA to tcl/tk, from LabVIEW to Visual Basic. DM is able to update about 2000 channels per second.
ALH is the alarm handler providing alarm viewing, logging and annunciation. It can handle up to 500 alarm condition changes per second.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

14. Pigi Paolucci - I.N.F.N. of Napoli
BaBar Alarm Handler
Alarm limit and severity are defined at database level;
The three structure is defined using a configuration file.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

15. Pigi Paolucci - I.N.F.N. of Napoli
BaBar Alarm Structure
Each EPICS channel has 2 alarm limits:
The first one is just a warning (color code is yellow);
The second one is a real alarm (color code is red).
The alarms are sent from the EPICS database to the EPICS Screen, Alarm Handler, Java Browser and Database.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

16. BaBar Data tacking philosophy
Sub-detectors are divided in logical components, called partitions, based on the hardware structure of the system.
Muon is divided in Barrel, Endcap forward and backward;
DIRC and EMC are divided in 12 sectors;
Each partition has its own EPICS records, State Machine and Alarm threes;
The partition concept has been introduced in order to have the possibility to take data with sub-parts of the detectors.
A Main State Machine has to coordinate the partitions and define-calculate-monitor a RUNNABLE , INJECTABLE or CALIBRATION flag as STATUS of that detector.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

17. Pigi Paolucci - I.N.F.N. of Napoli
Interface with PEP-II
Close communication between BaBar and PEP-II is essential to maintaining high operational efficiency and low background conditions.
Before injection can begin BaBar must be in safe state and it remains safe until injection is finished.
This injection sequence is coordinate automatically by two state machines, one running on the BaBar IOC verifies the detector status and communicate it to a second process running on the PEP-II IOC.
Special hardware button is used to prevent/stop injection from the BaBar console.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

18. Pigi Paolucci - I.N.F.N. of Napoli
BaBar State Machine
The RUNNABLE flag corresponds to a state in which the detector is ready to take data  HV at nominal voltage.
The INJACTABLE flag means that detector is safe for the PEP II injection  Tracker, Drift and Muon at safe voltage.
The CALIBRATION flag is used if one or more detectors are ready to take special run dedicated to the calibration.
The BaBar State Machine get all the flags from the detector state machines and analyze them in real time generating a BaBar flag (RUNNABLE , INJECTABLE or CALIBRATION), monitored in the main panel and sent to the PEP II panel and state machine
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

19. Pigi Paolucci - I.N.F.N. of Napoli
BaBar main panel
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

20. Muon system main panel

21. Wiener crate panels

22. Pigi Paolucci - I.N.F.N. of Napoli
Muon
HV Console
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

23. BaBar DCS architecture (overview)
Detector
EPICS
Control layer
PEP II
Database
Run Control
Component
Proxy
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

24. BaBar DCS architecture (details)
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

25. Pigi Paolucci - I.N.F.N. of Napoli
Component Proxy
A dedicated object-oriented interface, called Component Proxy, have been developed to bridge the gap between EPICS, written in C, and the BaBar Online system, implemented in C++.
Component Proxy design:
Abstraction of BaBar hardware in terms of C++ objects;
Communicates with Run Control, configure detector and archives data;
Generate the RUNNABLE flag of each subsystem.
BaBar runs 27 CP on two Sun Ultra 10 workstations with 0.74GB of memory.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

26. Interface with Run Control
Each logical component of BaBar (tracker, DIRC...) is represented by one or more instances of the Component Proxy.
Each proxy reports the status of its associated hardware components to the Run Control in form of a RUNNABLE flag, defined at EPICS level (state machine).
Run Control is not allowed to begin a data run until all components are RUNNABLE.
If a system loses its RUNNABLE status the Run Control pause the run.
The response feedback mechanism is typically < 1 sec.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

27. Pigi Paolucci - I.N.F.N. of Napoli
Ambient Database
BaBar has adopted an object oriented approach for its offline and online software.
Objectivity/DB is the underlying technology
Provides the storage of ambient data from the DCS
Design similar to Conditions Database
Operational information:
27 archiving processes write about 100 objects/hour;
Total size of objects about 3 MB;
Storing time about 1min;
Data written every 60 min to reduce DB activity;
Total rate of about 2 MB/hr (17.5 GB/yr).
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

28. Pigi Paolucci - I.N.F.N. of Napoli
Ambient Database
Java
Browser
Java
Browser
Ambient
Database
Ambient
Server
(C++/CORBA)
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

29. Configuration Database (I)
Provides storage for the configuring parameters of the online system;
Interface to Dataflow ROMs (DAQ) via UNIX server;
Configuration objects indexed on numeric keys;
Hierarchical tree-like structure;
Tree provides navigation from top, indexed by configuration key, to the subsystem configuration data on the leaves of the tree;
Simplifies system partitioning. Allows authorization control on subsystem level;
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

30. Configuration Database (II)
Before data-taking begins, Run Control initiates a CONFIGURE transition by sending a configuration key (defined by the run type) to the DAQ, trigger and DCS systems.
The key is used by the systems to read data from the Configuration database.
In the case of DCS the key is received by the Component Proxy then read data (typically HV set-points) and write it into EPICS.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

31. Configuration Database (III)
The system allows for easy switching between run types (physics, cosmic, calibration...) and create an archive of detector configurations in the database, used to determine the running conditions at any time.
Processes accessing the database with each configure transition:
32 Event Processing nodes
17 Detector Control processes
Dataflow server
Configuration time is about 15 sec
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

32. Pigi Paolucci - I.N.F.N. of Napoli
Conclusion (I)
The BaBar Detector Control is working since the 1999.
The EPICS core infrastructure has proven to be very robust, serving about 105 data channel over 15 IOCs in a very heavily used network
The communication between BaBar and PEP-II has also been very reliable, and has contributed to a high overall operating efficiency.
EPICS proved to be a very powerful, reliable and friendly system to control and monitor both BaBar and PEP II.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

33. Pigi Paolucci - I.N.F.N. of Napoli
BaBar Efficiency
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

34. Pigi Paolucci - I.N.F.N. of Napoli
Conclusion (II)
Implementation of the Component Proxy and ODC interface into the Online system has not been easy.
We had some problems with operational efficiency and scalability of the Objectivity database.
The locking mechanism that regulates reading and writing create conflicts between different online processes.
A big step has been produced from the DCS of the LEP experiments and the BaBar DCS system.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli

35. Pigi Paolucci - I.N.F.N. of Napoli
What we can improve !
Design a DCS upper layer for control and monitor detectors, accelerator and the services (gas, safety...);
A more friendly alarm handler with more and more information about the way to fix the problems;
A way to correlate data with the Ambient database and Configuration database information;
An easy way to debug the detector problem starting from one or more alarms/warning.
Automatic way to fix the more frequent error/problem.
16 November 2018
Pigi Paolucci - I.N.F.N. of Napoli