1. «The past experience with JCOP in ALICE» «JCOP Workshop 2015» 4-5 November 2015
Alexander Kurepin

2. The ALICE experiment at CERN
Collaboration:
37 countries
154 institutes
1500 members
The Detector:
tons
19 subdetectors built on different technologies
2 magnets
The ALICE experiment is the one of the 4 big experiments on LHC
In ALICE experiment involved about 1 thousand 5 hundred members from 157 institutes from 37 countries
Detector represents facility with weight about 10 thousand ton with 19 subdetectors, one solenoid magnet and one dipole magnet
The main goal of the experiment is studying havy ion collisions and research quark-gluon plasma. ALICE experiment also has proton-proton physics program.
Main physics focus:
Heavy ion physics at LHC energies
Proton-proton physics

3. ALICE Detector Control System (DCS)
Guaranties:
24/365 safe and stable operation of the experiment
Partial detector control system are built by experts of the detector teams
The Central DCS Team provides:
Distributed control system based on commercial components
Computing and network infrastructure
Software framework and tools
Implementation guidelines and operational rules
Integration of detector systems
Interfaces to external systems and LHC
Training of operators and uninterrupted expert on call service
First line support for software developers
ALICE DCS provide 24 hours every day of the year safe and stable operation of the experiment
Partial detector control system are built by experts of the detector teams
The Central DCS Team provides:
Distributed control system based on commercial components
Computing and network infrastructure
Software framework and tools
Implementation guidelines and operational rules
Integration of detector systems
Interfaces to external systems and LHC
Training of operators and uninterrupted expert on call service
First line support for software developers

4. The ALICE DCS – Context and Architecture
User Interface,
External Systems Interface
ALICE Framework
CERN JCOP Framework
SCADA System
(WINCC OA)
Devices with similar functionality are grouped into subsystems. About 100 different subsytems are implemented in ALICE.
Central DCS project controls all experiment subsystems.
For distributed control system ALICE DCS uses commercial SCADA system WinCC OA (open architecture) and other commercial products like OPC servers. To extend functionality in the project integrated JCOP framework. Together with ALICE and the detector experts panels this forms complete control project.
Device Interface Layer
(OPC or custom)
Devices
(Industrial and custom)

5. ALICE DCS Systems DETECTOR DETECTOR DETECTOR WCC OA WCC OA WCC OA
For each detector we could have several projects connected to the detector common distributed network.
The central systems connects to detector system and form one large distributed system sharing the data and the control.
WCC OA
WCC OA
The central systems connects to detector system and form one large distributed system sharing the data and the control
WCC OA
WCC OA
WCC OA
WCC OA

6. JCOP framework in ALICE
JCOP is an excellent example of successful common project at CERN
Without JCOP we would have hard times to develop the systems
ALICE would not be able to implement and run a DCS system as it is running now, or only at the expense of additional resources
JCOP is an excellent example of successful common project at CERN
Without JCOP we would have hard times to develop the systems
ALICE would not be able to implement and run a DCS system as it is running now, or only at the expense of additional resources
>As always happen with good products, people start to expect much more

7. ALICE challenges
We all depend on WinCC OA and Siemens support has rather high threshold to start solving the problems
Many components are very complex, but they lack documentation
Sometimes extremely difficult to debug code without having powerful tool for analyse log files
We often imagine a component differently from how it is seen by the developer, so possibility of customisation would be very helpful
Every time we are doing upgrade we always have to find compromise between OS and components
We can’t cay that we are running all the time without any problem. You will not believe.
We all depend on WinCC OA and Siemens support has rather high threshold to start solving problems
Many components are very complex, but they lack documentation. Often documentation are very basic.
>It would be great to have: All documentation available, as for many our colleagues it is quite difficult to attend JCOP trainings
Sometimes extremely difficult to debug code without having powerful tool for analyse log files.
>Correlation between 2 log files, to find exact event in different sources. During operation period to see what command was send between lots of FSM logs.
We would like to have mechanism to provide own experiment customisation and have them distributed in framework
AES as an example of confusing filtering mechanism
We would love to have the actual operating systems supported in time by all components. Usually there is always some showstopper and we need to go for compromise.

8. FSM in ALICE
FSM is very powerful instrument but at a few occasions we have suffered from an loop error in the logic of a detector blocking the whole DCS
One thing would be to implement a protection for loops
Another thing would be to have a (light) procedure to check the logic of the FSM
FSM is very powerful instrument but at a few occasions we have suffered from an loop error in the logic of a detector blocking the whole DCS
One thing would be to implement a protection for loops
Another thing would be to have a (light) procedure to check the logic of the FSM
We once did such an exercise, but that was a rather heavy procedure. It would be worth to put some effort to revive this in a lighter form.

9. ALICE DCS ideas Organize ALICE specific training on JCOP tools
In a collaboration of ALICE DCS team and EN/ICE
Organise training/testing laboratory. DCS team could offer it’s lab and infrastructure and can host devices provided by others (EN/ICE) for specific tests
Create ad-hoc working groups of experts from all experiments that can go into more technical details of a given subject and propose solutions/improvements and sharing information to the JCOP CB
Before we start the upgrades we could make a list of all components including the OS, look at the roadmaps and set a goal in order to have all at the same level when we starts an upgrade
We could offer some resources to look into FSM logic check
JCOP trainings specific for ALICE experiment needs.
In a collaboration of ALICE DCS team and EN/ICE
Organise training/testing laboratory. DCS team could offer it’s lab and EN/ICE could provide some rare specific devices
Create working group of experts and that can go into more technical details of a given subject and propose solutions/improvements and sharing information to the JCOP CB
Before we start the upgrades we could make a list of all components including the OS, look at the roadmaps and set a goal in order to have all at the same level when.
>Last time we missed the chance to go for the latest Windows server because of OPC.
We could offer some resources to look into FSM logic check

10. Detector experience
All ALICE detectors use JCOP framework in their projects
Number of components depends on needs
Very useful tool to prepare own FW component
All ALICE detectors use JCOP framework in their projects
Number of components depends on needs
Very useful tool to prepare own FW component

11. Useful tools for detector experts
For expensive devices to debug panels and scripts and teach newcomers it’s great to have powerful simulator
We have one for CAEN devices during the training lessons, but it is difficult to find
For expensive devices to debug panels and scripts and teach newcomers it’s great to have powerful simulator
We have one for CAEN devices during the training lessons, but it is difficult to find

12. ALICE operational model
We currently run with 4 shifters and on-call for all detectors and central systems (DCS on-call covered by central team)
In the past we have been running with 6 shifters (HLT, DCS, DAQ/ECS, CTP, DQM, SL)
We typically run the DCS shifts with a pool of ~100 different people to cover all shifts of a year
We currently run with 4 shifters and on-call for all detectors and central systems (DCS on-call covered by central team)
In the past we have been running with 6 shifters (HLT, DCS, DAQ/ECS, CTP, DQM, SL)
We typically run the DCS shifts with a pool of ~100 different people to cover all shifts of a year

13. ALICE DCS operation The DCS is operated by a single operator
In 2015 more than 150 operators were trained to assure the 24/7 operation
Most of the operators did not have prior experience with the DCS
We have developed dedicated panel that allows to perform routine actions with a single click (handshake, going safe at end of fill, going ready at start of fill etc.)
The interface to the DCS is based on set of intuitive graphics interfaces
From a single screen, the operator can access all controlled parameters
The DCS is operated by a single operator
In 2015 more than 150 operators were trained to assure the 24/7 operation
Most of the operators did not have prior experience with the DCS
We have developed dedicated panel that allows to perform routine actions with a single click (handshake, going safe at end of fill, going ready at start of fill etc.)
The interface to the DCS is based on set of intuitive graphics interfaces
From a single screen, the operator can access all controlled parameters

14. Shifts model
For RUN_2 period we changed the shifts blocks from 6 shifts in the same slot, to (MMAANN)
Participating shifts give to institutes the ‘credits’
This steps allow us to fill the shift crew more efficient
For RUN_2 period we changed the shifts blocks from 6 shifts in the same slot, to (MMAANN)
Participating shifts give to institutes the ‘credits’
This steps allow us to fill the shift crew more efficient

15. The ALICE O2 Project
LHC will provide for RUN3 ~100x more Pb-Pb collisions compared to RUN2
(1010 – collisions/year)
ALICE O2 Project merges online and offline into one large system
TDR approved!
Some O2 challenges:
New detector readout
8400 optical links
Data rate 1.1TB/s
Data storage requirements: ~60PB/year
LHC will provide for RUN3 ~100x more Pb-Pb collisions compared to RUN2
(1010 – collisions/year)
ALICE O2 Project merges online and offline into one large system
TDR approved!
Some O2 challenges:
New detector readout
8400 optical links
Data rate 1.1TB/s
Data storage requirements: ~60PB/year

16. DCS in the context of O2 DCS provides input to O2
~ conditions parameters are requested
Data has to be injected into each data frame at 20ms rate
DCS provides input to O2
~ conditions parameters are requested
Data has to be injected into each data frame at 20ms rate

17. Conclusion
Using JCOP components the DCS provides stable and uninterrupted services to the ALICE experiment since 2007
Modular and scalable design minimized the need for system modifications and coped with the evolution of hardware
The more we are using JCOP tools the more support we will need
Challenging O2 project requires full redesign of ALICE DCS data flow
Using JCOP components the DCS provides stable and uninterrupted services to the ALICE experiment since 2007
Modular and scalable design minimized the need for system modifications and coped with the evolution of hardware
The more we are using JCOP tools the more support we will need
Challenging O2 project requires full redesign of ALICE DCS data flow