1. Shift Leader Tutorial Made by : Maria Chamizo Andrea Barisone
11/13/2018
Shift Leader Tutorial
Made by :
Maria Chamizo
Andrea Barisone
Last Revision : September 24 , 2012
11/13/2018
Shift Leader Training Meeting

2. Shift Leader Training Meeting
11/13/2018
Outline
Become a Shift Leader
Main Responsibility
Documentation
Shift Leader : Location and Tasks
Monitoring Tools
Overview of LHC (Modes, Fill-Cycle)
Shift Leader Check List
Handshake
Injection and Injection Inhibit
Ramp, Flat top, Squeeze and Adjust
Basic Checks on Stable Beam
Pixel and Strip HV Procedure
BRM Displays
Adjust Handshake
End of Fill
Cosmic Run
Magnet Ramping
Conclusion
11/13/2018
Shift Leader Training Meeting

3. Shift Leader Training Meeting
11/13/2018
Become a Shift Leader
To become a full operational Shift Leader remember you must follow :
Shift Leader Tutorial (ok , which we are doing now !! )
DCS Tutorial (two Wednesday each month at 14:00 , P5 meeting room)
SLIMOS Tutorial
Before do alone his first Shift every new SL will be scheduled for 2 training Shift.
Please contact Andrea Barisone
to be signed up :
11/13/2018
Shift Leader Training Meeting

4. Main Responsibilities
11/13/2018
Main Responsibilities
The Shift Leader is also the SLIMOS (Shift Leader in Matter of Safety)
You will discover all the details about the SLIMOS duties in the specific tutorial, at the moment, just remember that you are responsible for the SAFETY of :
People
Experiment
2) As a Shift Leader you’re the Reference for the CMS Control Room Activities :
The SL coordinates the implementation of the daily run plan in the most efficient way
The SL is the contact person between CMS and LHC control room (CCC)
Important note : In case of doubt, he must call the Run Field Manager
(75575) or Run Coordinator ( )
11/13/2018
Shift Leader Training Meeting

5. Shift Leader Training Meeting
11/13/2018
Documentation (1)
Much more detailed descriptions about the SL task are available in the dedicated online page, the “Shift Leader Duties and Instructions” : ( inside the CMS Workbook, here below the link)
It contains an exhaustive description of:
Shift Leader General Information
 Safety and Technical Interventions
Instructions Associated to LHC
Instructions Associated to Magnet Operations
Tools (Elog, Run Time Logger, Online Monitoring Site)
11/13/2018
Shift Leader Training Meeting

6. Shift Leader Training Meeting
11/13/2018
Documentation (2)
Important note : At the beginning of the Shift, the SL has to verify first of all the breaking news : You will find them on the top of the “SL Duties and Instructions” page.
11/13/2018
Shift Leader Training Meeting

7. Shift Leader Training Meeting
11/13/2018
Documentation (3)
Another really important document is the Contacts List :
Where you will find all the telephone numbers you could need to
call during your shift !
11/13/2018
Shift Leader Training Meeting

8. Shift Leader : Location and Task (1)
11/13/2018
Shift Leader : Location and Task (1)
Control Room Overview
DAQ Shifter
DCS Shifter
DQM Shifter
Trigger Shifter
This is the Shift Leader Place
BRM
Displays
Injection Button
11/13/2018
Shift Leader Training Meeting

9. Shift Leader : Location and Task (2)
11/13/2018
Shift Leader : Location and Task (2)
SL Station Overview
LHC main screen
Tandeberg (Videoconference)
Events Displays
SL Operational Displays
11/13/2018
Shift Leader Training Meeting

10. Shift Leader : Location and Task (3)
11/13/2018
Shift Leader : Location and Task (3)
Phones:
There are two phones at the Station of the SL
Mobile phone mapped to the virtual phone number This is used by the CCC or people outside CMS to contact the CMS control room.
Cordless phone: a fixed line (77111 )to receive calls made by CMS members.
Remember, the most relevant contacts are available here:
Important: Mobile calling SHOULD ALWAYS BE ANSWERED                   If you carry this phone with you ... don't forget to leave it there at the
end of your shift.
11/13/2018
Shift Leader Training Meeting

11. Shift Leader : Location and Task (4)
11/13/2018
Shift Leader : Location and Task (4)
Communication :
An important part of the SL job is a good and clear communication between him\her and the Shift Crew, that means :
Interact with the other Shifter (DCS,DAQ,DQM and Trigger) to know what is going on
Decide (or be aware) when an expert on-call must be contacted
Filter the subsystem request in order to preserve the Run
Be the reference for the LHC Control Room (CCC)
Many – but not all – DQM shifters are relatively inexperienced
As shift leader we expect you to interact with the DQM shifter
In particular: at start of new runs work with the DQM to make sure
that there are no problems.
11/13/2018
Shift Leader Training Meeting

12. Shift Leader Training Meeting
11/13/2018
Monitoring Tools (1)
There’s one simple question that could be come in your mind, this one :
“Which are the the means at my disposal to fulfill my responsibilities?”
First of all, you should open the CMS Run Coordination Page :
In this section you will find the Shift Leader Instructions and the Tutorial
On his left, there are listed all the main tools to check the CMS activity, in the next pages we will
explore some of them.
Here there is the Phone List … discover by yourself all the other links!
11/13/2018
Shift Leader Training Meeting

13. Shift Leader Training Meeting
11/13/2018
Monitoring Tools (2)
Run Time Logger (RTL) :
The RTL is used to monitor the downtime/lifetime status of CMS during data taking. It automatically creates a downtime entry (provided SCAL FM is in the readout) when there are no L1 triggers in CMS. The SL has to choose which subsystem is responsible for the downtime and report it in the RTL.
11/13/2018
Shift Leader Training Meeting

14. Shift Leader Training Meeting
11/13/2018
Monitoring Tools (3)
E-log:
The SL is responsible for entering information in the shift e-log to describe :
Start/End of the shift / Data taking conditions/problems / Hourly updates (during the Run)
11/13/2018
Shift Leader Training Meeting

15. 11/13/2018
Monitoring Tools (4)
You can check the expected status of every CMS subdetectors (according to the LHC beam mode) looking to the Automation page.
You can find it in the CMS Run Coordination page as “ Detector Safety Mode”
Or directly clicking on this link :
11/13/2018
11/13/2018
Shift Leader Training Meeting
Shif Leader Training Meeting 15

16. Shift Leader Training Meeting
11/13/2018
Monitoring Tools (5)
At your station, in front of you, little above your main displays, there’s a big screen where are reported all the relevant information about LHC status : We will discover later how to get quickly the main information you often need to check. As usual you have a link at these pages also in the Run Coordination Page.
11/13/2018
Shift Leader Training Meeting

17. Shift Leader Training Meeting
11/13/2018
LHC : Diagram
11/13/2018
Shift Leader Training Meeting

18. Shift Leader Training Meeting
11/13/2018
LHC : Modes
The LHC modes are summary of the machine state.
Three main Modes are defined :
Accelerator Mode =summary of the LHC machine state
Beam Mode = description of the main phases of the accelerator cycle
Sector Operation Mode = is a given LHC sector operational or not? ( We won’t consider this part in our Tutorial)
Next two pages contains a more exhaustive description about the first two Modes :
We leave you alone to read and if you like get really detailed information download the following document inside the EDMS Portal
11/13/2018
Shift Leader Training Meeting

19. LHC Accelerator Mode Accelerator mode Description Beam Status
11/13/2018
LHC Accelerator Mode
Accelerator mode
Description
Beam Status
SHUTDOWN
Usual winter status. Cold magnets floating.
NO BEAM
COOLDOWN*
Coming back from shutdown. The principal activity during this mode is cryogenics related
MACHINE CHECKOUT*
Check the simultaneous functioning of the various LHC sub-systems in the final configuration following the LHC hardware commissioning
ACCESS*
Access
MACHINE TEST*
Operations’ tests without beam. Ad hoc tests without beam during normal running periods
CALIBRATION*
Power Converter calibration. No external interference with circuits
WARM UP*
One or more sectors warming up for repair.
RECOVERY*
Typically quench recovery, or recovery from cryogenics plant disturbance.
SECTOR DEPENDENT
This Accelerator Mode tries to cope with the situation in which different sectors of the machine can be in different states at the same time
BEAM SETUP
Machine setup with one or both beams
BEAM
PROTON PHYSICS
Beam based operation aimed at proton physics.
ION PHYSICS
Beam based operation aimed at ion physics.
SPECIAL OPTICS PHYSICS
Beam based operation with special optics, like for example the one for TOTEM.
MACHINE DEVELOPMENT
Beam based machine development.
11/13/2018
Shift Leader Training Meeting

20. LHC : Beam Mode Beam mode Description Beam mode Description 11/13/2018
SETUP
Possibly beam in transfer lines with transfer line dumps in. Includes pre-injection plateau and injection plateau - no beam in ring.
ABORT
Recovery mode following beam permit drop. This mode can be entered from any state if there is no beam in the machine.
INJECTION PROBE BEAM
If either ring 1 or ring 2 will be injected with or have safe beam circulating. In this mode a number of checks will be done for the different accelerator sub-systems before injecting higher intensities. The aim will be to establish a circulating safe beam with a given lifetime
INJECTION SETUP BEAM
In order to make more precise measurements before filling for physics, a SETUP BEAM will be used.
INJECTION PHYSICS BEAM
At this stage the machine has been optimized. It proved to be able to have circulating beam with appropriate lifetime and it is ready to accept higher intensities needed for physics
PREPARE RAMP
Injection complete, preparing for ramp.
RAMP
Ready to ramp or ramping or immediate post ramp.
FLAT TOP
Ramp finished - pre-squeeze checks.
SQUEEZE
Preparing for or squeezing.
ADJUST
Preparing for collisions or adjusting beams after the squeeze.
Beam mode
Description
STABLE BEAM
Stable conditions with collisions in the experiments, backgrounds and life time under control. Small adjustment of beam parameters permitted.
UNSTABLE BEAM
Emergency mode entered from stable beams in case of sudden beam degradation. The UNSTABLE BEAMS mode may be entered without prior warning to the experiments
BEAM DUMP
Requested or emergency dump. It will be verified that all the machine protection equipment performed correctly
RAMP DOWN
Ramp down and cycling after programmed dump at end of physics fill.
CYCLING
Pre-cycle before injection following access, recovery, etc. The objective of this mode is to reset the magnetic history of the machine and prepare the machine for a new cycle.
RECOVERY
Within this mode the reason of the abort, quench, emergency dump, etc, will be diagnosed by the post mortem analysis system.
INJECT AND DUMP
This mode may be used during first commissioning or for injection studies. I
CIRCULATE AND DUMP
Dump after large number of turns following injection
NO BEAM
In a machine mode where there is no beam or no preparation for beam.
11/13/2018
Shift Leader Training Meeting

21. Shif Leader Training Meeting
11/13/2018
LHC : Fill Cycle
Flat Top
11/13/2018
Shif Leader Training Meeting

22. LHC : Beam Mode (Sequence)
11/13/2018
LHC : Beam Mode (Sequence) 1 3 2 1 10
Start of a Cosmic Run
Handshake
(Injection)
LHC
CMS
Stop the Cosmic Run 9 2
Inhibit Injection 3 4 5 6 7 8
Start of a Physic Run
11/13/2018
Shift Leader Training Meeting

23. Shift Leader Training Meeting
11/13/2018
LHC : Get Information
The best way to be in touch with the LHC operation is checking regularly its Elog :
11/13/2018
Shift Leader Training Meeting

24. Shift Leader Check List
11/13/2018
Shift Leader Check List
To help the Shift Leader we maintain a detailed checklist that he must follow
during his shift.
11/13/2018
Shift Leader Training Meeting

25. How to use the Check List
11/13/2018
How to use the Check List
It’s frequently updated to reflect latest changes so should be the most up-to-date document : Always make sure you are using the latest version !!!
(print it from to the SL instruction page , on the right of the Breaking News window)
As an LHC fill can cover several shifts, please continue the same checklist, but note where you have take over. At the end of the fill put the checklist into the red binder.
If you find problems with the checklist please inform the RFMs or the run
coordinator. No “oral traditions” are allowed.
Take care : there are several different checklists
( Proton Physics 8TeV,Heavy Ions, Machine Development … )
so chose the right one !
11/13/2018
Shift Leader Training Meeting

26. Shift Leader Training Meeting
11/13/2018
Handshake
The Handshake Operation :
Before LHC issues the Injection Warning handshake (seen in DCS console) you should be contacted by LHC control room (CCC) : they will ask you to lift the beam injection inhibit button.
Once performed all the steps in your Check List dedicated to this operation you must pull the injection inhibit button to allow LHC to inject.
Here you can see the part of the DCS Main panel dedicated to the Handshake operation : once pulled the injection button this image should change showing its released form
11/13/2018
Shift Leader Training Meeting

27. Shift Leader Training Meeting
11/13/2018
Injection Inhibit
Injection inhibit.
Green light off -
injection inhibited
Beam abort
11/13/2018
Shift Leader Training Meeting

28. Shift Leader Training Meeting
11/13/2018
Injection (1)
When LHC issues Injection Warning handshake (seen in DCS console):
Detectors will automatically be brought to a safe state
If pixels or strips are off – call the pixel and strip DOCs
Stop the Cosmic Run
Once all checks in the checklist are performed the SL pulls the injection inhibit button to allow LHC to inject. (PAY ATTENTION OF DOESN’T ACT ON THE WRONG BUTTON, THE BEAM ABORT !!)
LHC leaves setup beam mode and goes to injection probe beam
No major action required (follow the Check-List)
Then LHC changes mode again to Injection Physics Beam.
11/13/2018
Shift Leader Training Meeting

29. Shift Leader Training Meeting
11/13/2018
Injection (2)
After injection is complete LHC goes to Prepare Ramp mode
Shift leader pushes the injection inhibit button to prevent further injections (PAY ATTENTION OF DOESN’T ACT ON THE WRONG BUTTON, THE BEAM ABORT !!) then LHC announces on LHC page 1 that next fill is for physics
Verify that CMS inhibits injection (green light off - button pushed in)
Follow checklist to make sure all subsystems are ready.
Since the beginning of 2012 if the injection is allowed Pixels and Tracker will not be able to switch on the HV.
LHC Accelerator Mode should be proton physics
11/13/2018
Shift Leader Training Meeting

30. Ramp, Flat Top, Squeeze and Adjust
11/13/2018
Ramp, Flat Top, Squeeze and Adjust
Ramp:
No major action required (follow the Check-List)
Flat Top:
Start a Physics Run
Make sure we are using LHC clock – check with DAQ
Check the cogging
Ask DAQ shifter to destroy RCMS configuration and recreate it. Start a run with the collision configuration and check that the correct combination of keys are used.
Squeeze and Adjust :
At Squeeze DCS automatically turns HV for ZDC, DT, and RPC.
In case the LHC skips the Squeeze mode the DCS shifter should
manually turn on HV for DT and RPC.
11/13/2018
Shift Leader Training Meeting

31. Shift Leader Training Meeting
11/13/2018
Stable Beam
Stable Beam:
Immediately after LHC has declared the Stable Beam the Shift Leader and the DCS shifter check that the beam conditions are acceptable to turn ON the Pixel and Strip HV (more details in the next page).
When HV is ramped for strips and pixel the DAQ automatically do a Pause & Resume.
DAQ receives the PHYSICS_ON flag.
If the LHC does not perform a miniscan shortly after stable beams call CCC and
request a scan.
Also if CMS lumi is more than ~5% lower than ATLAS request a lumi scan.
11/13/2018
Shift Leader Training Meeting

32. Pixel and Strip HV Procedure (1)
11/13/2018
Pixel and Strip HV Procedure (1)
When the beam mode reaches STABLE BEAMS with machine mode of PROTON PHYSICS, the Shift Leader and DCS shifter should check the state of the Pixel and Strips detectors and be sure that they have automatically turned ON.
If after ~1.5 minutes of STABLE BEAMS the Pixel and Strips detectors have NOT turned to 100% ON and/or are stuck in GOING_TO_PHYSICS state:
DCS shifter send the GO_TO_PHYSICS command once (to the corresponding partition)
if sending the GO_TO_PHYSICS command once does not bring the specific sub-detector ON, the Shift Leader must call the corresponding Pixel and /or Strips DOC.
If they stay in STANDBY, the automated turn-on did not work, maybe because the backgrounds were not OK when Stable Beams were declared. The DCS shifter follow the instructions to manually turn ON the Tracker (see next page)…
Important: the communication between the DCS shifter and Shift Leader is crucial for good cross checking !
11/13/2018
Shift Leader Training Meeting

33. Pixel and Strip HV Procedure (2)
11/13/2018
Pixel and Strip HV Procedure (2)
If after ~1.5 minutes of STABLE BEAMS (machine mode of PROTON PHYSICS ) the Pixel and Strips detectors are still in STANDBY the Shift Leader ( with the support of the DCS shifter ) checks if the beam conditions are acceptable to manually turn ON the Pixel and Strip HV.
Here is the check list with the controls that must be performed (the main part):
INJECTION must be inhibited
BKGD1 and BKGD2 must be below 20
BKGD3 must be below 40
BEAM MODE must be STABLE BEAMS
When the conditions for turning on the HV are met, the DCS shifter turns ON their HV Channels. (remember : he should do that only after your agreement )
The complete procedures are available in the usual spot:
If any of these conditions are not met call BRM and Pixel
Since 2012 if the injection is allowed (the injection inhibit button is released and its green light is on) Pixels and Strip Tracker won’t be able to switch on their HV (as additional security measure) .
11/13/2018
Shift Leader Training Meeting

34. Shift Leader Training Meeting
11/13/2018
BRM Displays
SCX5SCR27-CMS
SCX5SCR28-CMS
Top Left
BRM 1hr Summary
Middle Left
BCM
Bottom Left
BSC/BCM1F
&
DIP browsers
Top Center
Shift leader display
Middle Center
BPTX scope snapshots?
BCM1F bunch ID?
Top Right
Lumi Display
Middle Right
LHC Bunches
Bottom Right
Post Mortem
11/13/2018
Shift Leader Training Meeting

35. Shift Leader Training Meeting
BRM Screen
11/13/2018
Luminosity
CMS Rates
Currents and Energy
Bunches
BKGDS sent to LHC
Timing
BCM.
BCM.
11/13/2018
Shift Leader Training Meeting

36. Basic Checks to do during Stable Beam (1)
11/13/2018
Basic Checks to do during Stable Beam (1)
Record in the Elog at the beginning and every hour:
(Initial) beam intensity per beam
Instantaneous luminosity
Background levels for BKGD1, BKG2 and BKG3
Total L1 rate (you can ask the Trigger shifter)
Stream A rate
Express Stream rate
Stream A Event size
DQM status (check done with the DQM shifter)
Trigger status (check done with the Trigger shifter)
DAQ status (check done with the DAQ shifter)
11/13/2018
Shift Leader Training Meeting

37. Basic Checks to do during Stable Beam (2)
11/13/2018
Basic Checks to do during Stable Beam (2)
Backgrounds Levels
Instantaneous Lumi
Initial Beam Intensity
11/13/2018
Shift Leader Training Meeting

38. Basic Checks to do during Stable Beam (3)
11/13/2018
Basic Checks to do during Stable Beam (3)
Lv1 Rate
Event Size
Stream A
Express Rate
11/13/2018
Shift Leader Training Meeting

39. Basic Checks to do during Stable Beam (4)
11/13/2018
Basic Checks to do during Stable Beam (4)
Here you can see the State of each Subdetector and the number of FED involved in the data taken
Here you can see how many slices of the PC farm are in or out the data taken
11/13/2018
Shift Leader Training Meeting

40. Shift Leader Training Meeting
11/13/2018
Adjust Handshake
The LHC might want to go out of stable beams to make orbit adjustments etc.
This is done by them issuing an Adjust_Warning
DCS automatically sends pixels, strips, and ZDC to standby.
As shift leader you inform pixel and strip DOCs that an adjust handshake is ongoing.
When stable beams again declared follow the procedure to turn on the HV for
pixel and strips
11/13/2018
Shift Leader Training Meeting

41. Shift Leader Training Meeting
11/13/2018
End of Fill
When LHC issues the Beam Dump Warning :
Three minutes after the dump warning, DCS sends PIXEL, STRIP and ZDC automatically to STANDBY, if any of the them are ON. The DCS Shifter cross checks the sub-detector status with the Safe Mode Detector Automation.
The beams are automatically dumped five minutes after the Warning was sent. Continue the run until the beams have been dumped.
At this point a Cosmic Run (with possible calibration activities) can start.
11/13/2018
Shift Leader Training Meeting

42. Cosmic Run after Beam Dump
11/13/2018
Cosmic Run after Beam Dump
After the beams have been dumped we want to start a cosmic run as soon as possible in order to have continuous monitoring of the ECAL.
Reconfigure for cosmic data taking using the local clock.
Make sure to include RPCs, CSCs, and DTs in the run.
Confirm that CMS is inhibiting injection.
When LHC goes to beam mode RAMP DOWN:
The pixels HV will turn on automatically.
Ask C-DCS shifter to switch ON STRIP TRACKER
End the run when LHC starts the injection handshake (detectors will automatically go to standby).
11/13/2018
Shift Leader Training Meeting

43. 11/13/2018
Magnet Ramping
When the CMS solenoid is ramping several subsystems needs to be put into a safe state:
ECAL off
HCAL standby (HF can be on)
CSC off
DT standby
CASTOR off
Strips and pixels can be on.
As shift leader you should make sure that these detectors are in the off state
before any intentional ramp takes place.
11/13/2018
Shift Leader Training Meeting

44. Shift Leader Training Meeting
11/13/2018
Conclusions
Your most important tasks :
You are the SLIMOS (Shift Leader in matter of Safety)
You coordinate the implementation of the Daily Run Plan in the most efficiently way
You are the contact person between CMS and LHC control room (CCC)
At the start of your Shift, verify here the breaking news :
Interact with the other Shifter (DCS,DAQ,DQM and Trigger) to know what’s going on
Decide (or be aware) when an expert on-call must be contacted
Filter the subsystem request in order to preserve the Run
Your most powerful tools to achieve a good result:
Check List
Contact Phone List
Run Time Logger
Elog
Shift Leader Online Instructions
Run Coordination Page
11/13/2018
Shift Leader Training Meeting