1. Offline shifter training tutorial
L. Betev
July 23, 2009

2. The dashboard (see Costin’s talk)
Outline
Offline shifter basic responsibilities
The shifter check list
Systems and tools
The dashboard (see Costin’s talk)
The Shuttle (see Chiara’s talk)
The reonctruction and visualization package (see Marco’s talk)

3. Basic responsibilities – RAW data
The RAW data path
DAQ online
Fast optical link to CERN CC
500MB/sec (p+p), 1.25GB/sec (Pb+Pb)
Step A
CASTOR2 disk buffer
reduced
CASTOR2 tape buffer
Step B

4. Step A – Online buffer -> CASTOR buffer
Automatic and well-exercised (it almost never goes wrong)
At this step, the files are also registered in the AliEn catalogue
DAQ is nominally responsible for the transfers
Offline provides the registration gateway
If not working, DAQ notifies the shifter and/or the expert list
Offline monitors the fill of the CASTOR buffer (see dashboard)
The shifter will be responsible for copying of portions of RAW to tape (step B)

5. Step A – Shifter responsibilities
Monitors the fill of the CASTOR buffer (through the dashboard)
Notify the run coordinator/shift leader if more than 80% full
Follow the registration of RAW (through the dashboard)
All files in PHYSICS partition typically go to CASTOR
Follow the run screen and grow suspicious if none of the runs are being registered
Contact the DAQ shifter and ask what is going on

6. Step B – CASTOR buffer -> Tape storage
New this year – selective copying of runs to tape
1/5 of RAW data stream in p+p (100 MB/sec)
Full data stream in Pb+Pb (1.25GB/sec)
Exact procedure and decision path is being elaborated
It will involve some automatic copying (calibration data for example) and physics board/run coordinator decisions
The Offline shifter will be responsible for the copy procedure (though dashboard tools)
Also for the deletion of data from the CASTOR buffer

7. Basic responsibilities – Shuttle
Covered in Chiara’s presentation
Here just to put it in the context of the basic responsibilities

8. Basic responsibilities – fast reco and event display
A quick method to check the reconstruction of data and display couple of events from recent runs
NOT a tool to do analysis
Covered in Marco’s presentation
Here just to put it in the context of the basic responsibilities

9. Basic responsibilities – data replication
After RAW is recorded to tape in CASTOR2
A copy is made to a remote T1 centre for custodial storage (and processing)
The replication is an automatic process, triggered at EoR
Progress is displayed on the dashboard
Beginning of data taking – automatic replication is disabled
In general – the Offline shifter should follow the replication and raise alarm in case of failures

10. Basic responsibilities – prompt offline processing
After RAW is recorded to tape in CASTOR2 + Shuttle is done
Processing is launched
The processing is an automatic process
Progress is displayed on the dashboard
Beginning of data taking – automatic processing is disabled
Lists of runs to be processed is compiled by the run coordinator / shift leader

11. Basic responsibilities – prompt offline processing (2)
The experiment logbook contains ‘hints’ - run quality flags
Per detector and global
The run quality flags are presently filled manually, in the future by the Online QA
Offline shifter responsibility is to follow for all PHYSICS runs the content of the quality flags and prompt the shift leader and the detector shifters to fill these ate EoR

12. Offline shifter check list
Registration of RAW (dashboard)
Periodic check of status
Follow PHYSICS runs
Ask shift leader in case of doubt
Report registration errors to on-call expert
The run copy and removal procedure – to be defined
Shuttle (dashboard)
Follow on processing of all runs + global Shuttle messages
In case of preprocessor failures, escalate to (concerned) detector shifters
In case of Shuttle failures first follow the restart/debug procedures, then report to on-call expert

13. Offline shifter check list (2)
Fast reconstruction and event display (processing scripts on shifter console)
Periodic check of PHYSICS runs (not the entire run!)
Run reconstruction and analyse the AliRoot log files for errors/crashes
Note the above in the shifter report pages and send to
Visualize periodically events in PHYSICS runs
Note ‘strange’ event characteristics in the shifter report pages and send to

14. Offline shifter check list (3)
Data replication (dashboard)
Periodic check of replication status
Note ‘stuck’ runs – not replicated 12 hours after registration – in the shifter report pages and sent list to
Prompt data processing (dashboard)
Periodic check of processing status
Note ‘stuck’ runs – not processed 12 hours after registration – in the shifter report pages and sent list to
Shift report (shifter system)
At end of shift – summary of the operation and noteworthy events

15. General shifter rules
Before pressing the
Read the procedures and rules, defined for each error type
Try out the remedies
If all fails, inform the on-call expert

16. Information sources for the shifter
The shifter manual – instructions
Was here
Introducing the new Shifter interface
Monitoring – MonALISA
Dashboard
Shuttle
Processing and data management