Offline shifter training tutorial

Slides:



Advertisements
Similar presentations
CWG10 Control, Configuration and Monitoring Status and plans for Control, Configuration and Monitoring 16 December 2014 ALICE O 2 Asian Workshop
Advertisements

Clara Gaspar on behalf of the LHCb Collaboration, “Physics at the LHC and Beyond”, Quy Nhon, Vietnam, August 2014 Challenges and lessons learnt LHCb Operations.
June 19, 2002 A Software Skeleton for the Full Front-End Crate Test at BNL Goal: to provide a working data acquisition (DAQ) system for the coming full.
1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.
Grid and CDB Janusz Martyniak, Imperial College London MICE CM37 Analysis, Software and Reconstruction.
DØ Global Monitoring and Ideas for LHC/CMS remote monitoring Pushpa Bhat Fermilab.
O FFLINE T RIGGER M ONITORING TDAQ Training 30 July 2010 On behalf of the Trigger Offline Monitoring Experts team.
If you are very familiar with SOAR, try these quick links: Principal’s SOAR checklist here here Term 1 tasks – new features in 2010 here here Term 1 tasks.
Central DQM Shift Tutorial Online/Offline. Overview of the CMS DAQ and useful terminology 2 Detector signals are collected through individual data acquisition.
Central DQM Shift Tutorial Online/Offline. Overview of the CMS DAQ and useful terminology 2 Detector signals are collected through individual data acquisition.
Central DQM Shift Tutorial Online/Offline. Overview of the CMS DAQ and useful terminology 2 Detector signals are collected through individual data acquisition.
5. Data Manager 1. Introduction 2. Data Manager Duties 3. Quality Checking 4. Problem Reporting 5. Data Monitoring 6. Histogram Presenter 7. Trend Presenter.
Offline Tracker DQM Shift Tutorial. 29/19/20152 Tracker Shifts Overview Online Shifts at P5 (3/day for 24 hours coverage) – One Pixel shifter and one.
11 CTP Training A.Jusko, M. Krivda and R.Lietava..
ALICE Roadmap for 2009/2010 Patricia Méndez Lorenzo (IT/GS) Patricia Méndez Lorenzo (IT/GS) On behalf of the ALICE Offline team Slides prepared by Latchezar.
GGUS summary (7 weeks) VOUserTeamAlarmTotal ALICE ATLAS CMS LHCb Totals 1 To calculate the totals for this slide and copy/paste the usual graph please:
Costin Grigoras ALICE Offline. In the period of steady LHC operation, The Grid usage is constant and high and, as foreseen, is used for massive RAW and.
Offline shifter training tutorial L. Betev February 19, 2009.
Production status Preparation for HI running ALICE TF+AF November 14, 2010.
OFFLINE TRIGGER MONITORING TDAQ Training 5 th November 2010 Ricardo Gonçalo On behalf of the Trigger Offline Monitoring Experts team.
A.Golunov, “Remote operational center for CMS in JINR ”, XXIII International Symposium on Nuclear Electronics and Computing, BULGARIA, VARNA, September,
Offline report – 7TeV data taking period (Mar.30 – Apr.6) ALICE SRC April 6, 2010.
DQM status report Y. Foka (GSI) Offline week from pp to PbPb.
5/2/  Online  Offline 5/2/20072  Online  Raw data : within the DAQ monitoring framework  Reconstructed data : with the HLT monitoring framework.
Planning and status of the Full Dress Rehearsal Latchezar Betev ALICE Offline week, Oct.12, 2007.
CERN – Alice Offline – Thu, 20 Mar 2008 – Marco MEONI - 1 Status of Cosmic Reconstruction Offline weekly meeting.
Part I – Shifter Duties Part II – ACR environment Part III – Run Control & DAQ Part IV – Beam Part V – DCS Part VI – Data Quality Monitoring Part VII.
1 Checks on SDD Data Piergiorgio Cerello, Francesco Prino, Melinda Siciliano.
DQM for the RPC subdetector M. Maggi and P. Paolucci.
Prompt Calibration Loop 11 February Overview Prompt calibration loop in SCT –Provides ATLAS with conditions data used for the bulk reconstruction.
LCG-LHCC mini-review ALICE Latchezar Betev Latchezar Betev for the ALICE collaboration.
Status of the Shuttle Framework Alberto Colla Jan Fiete Grosse-Oetringhaus ALICE Offline Week October 2006.
Workflows and Data Management. Workflow and DM Run3 and after: conditions m LHCb major upgrade is for Run3 (2020 horizon)! o Luminosity x 5 ( )
1 Andrea Sciabà CERN The commissioning of CMS computing centres in the WLCG Grid ACAT November 2008 Erice, Italy Andrea Sciabà S. Belforte, A.
Predrag Buncic CERN ALICE Status Report LHCC Referee Meeting 01/12/2015.
Data processing Offline review Feb 2, Productions, tools and results Three basic types of processing RAW MC Trains/AODs I will go through these.
Online Consumers produce histograms (from a limited sample of events) which provide information about the status of the different sub-detectors. The DQM.
ALICE experiences with CASTOR2 Latchezar Betev ALICE.
Dynamic staging to a CAF cluster Jan Fiete Grosse-Oetringhaus, CERN PH/ALICE CAF / PROOF Workshop,
 offline code: changes/updates, open items, readiness  1 st data taking plans and readiness.
ALICE Physics Data Challenge ’05 and LCG Service Challenge 3 Latchezar Betev / ALICE Geneva, 6 April 2005 LCG Storage Management Workshop.
GGUS summary (3 weeks) VOUserTeamAlarmTotal ALICE7029 ATLAS CMS LHCb Totals
AAF tips and tricks Arsen Hayrapetyan Yerevan Physics Institute, Armenia.
ECAL Shift Duty: A Beginners Guide By Pourus Mehta.
SYSTEM INTEGRATION TESTING Getting ready for testing shifts Gunter Folger CERN PH/SFT Geant4 Collaboration Workshop 2011 SLAC.
THIS MORNING (Start an) informal discussion to -Clearly identify all open issues, categorize them and build an action plan -Possibly identify (new) contributing.
Federating Data in the ALICE Experiment
SchoolSuccess for Coordinators
Central DQM Shift Tutorial Online/Offline
WP18, High-speed data recording Krzysztof Wrona, European XFEL
Cross-site problem resolution Focus on reliable file transfer service
ALICE Monitoring
DPG Activities DPG Session, ALICE Monthly Mini Week
LHC experiments Requirements and Concepts ALICE
Status of the CERN Analysis Facility
ALICE analysis preservation
Patricia Méndez Lorenzo ALICE Offline Week CERN, 13th July 2007
Shift instructions August 16, 2017 Antoni Aduszkiewicz
Experience between AMORE/Offline and sub-systems
Data Preparation Group Summary of the Activities
Offline shifter training tutorial
Data Quality Monitoring of the CMS Silicon Strip Tracker Detector
Central DQM Shift Tutorial Online/Offline
QA tools – introduction and summary of activities
Offline monitoring, shifter dashboard
CMS Pixel Data Quality Monitoring
AIRWays Benchmark Previewing System
DQM for the RPC subdetector
CMS Pixel Data Quality Monitoring
Offline framework for conditions data
Presentation transcript:

Offline shifter training tutorial L. Betev July 23, 2009

The dashboard (see Costin’s talk) Outline Offline shifter basic responsibilities The shifter check list Systems and tools The dashboard (see Costin’s talk) The Shuttle (see Chiara’s talk) The reonctruction and visualization package (see Marco’s talk)

Basic responsibilities – RAW data The RAW data path DAQ online buffer @P2 Fast optical link to CERN CC 500MB/sec (p+p), 1.25GB/sec (Pb+Pb) Step A CASTOR2 disk buffer reduced CASTOR2 tape buffer Step B

Step A – Online buffer -> CASTOR buffer Automatic and well-exercised (it almost never goes wrong) At this step, the files are also registered in the AliEn catalogue DAQ is nominally responsible for the transfers Offline provides the registration gateway If not working, DAQ notifies the shifter and/or the alice-shift-alarms@cern.ch expert list Offline monitors the fill of the CASTOR buffer (see dashboard) The shifter will be responsible for copying of portions of RAW to tape (step B)

Step A – Shifter responsibilities Monitors the fill of the CASTOR buffer (through the dashboard) Notify the run coordinator/shift leader if more than 80% full Follow the registration of RAW (through the dashboard) All files in PHYSICS partition typically go to CASTOR Follow the run screen and grow suspicious if none of the runs are being registered Contact the DAQ shifter and ask what is going on

Step B – CASTOR buffer -> Tape storage New this year – selective copying of runs to tape 1/5 of RAW data stream in p+p (100 MB/sec) Full data stream in Pb+Pb (1.25GB/sec) Exact procedure and decision path is being elaborated It will involve some automatic copying (calibration data for example) and physics board/run coordinator decisions The Offline shifter will be responsible for the copy procedure (though dashboard tools) Also for the deletion of data from the CASTOR buffer

Basic responsibilities – Shuttle Covered in Chiara’s presentation Here just to put it in the context of the basic responsibilities

Basic responsibilities – fast reco and event display A quick method to check the reconstruction of data and display couple of events from recent runs NOT a tool to do analysis Covered in Marco’s presentation Here just to put it in the context of the basic responsibilities

Basic responsibilities – data replication After RAW is recorded to tape in CASTOR2 A copy is made to a remote T1 centre for custodial storage (and processing) The replication is an automatic process, triggered at EoR Progress is displayed on the dashboard Beginning of data taking – automatic replication is disabled In general – the Offline shifter should follow the replication and raise alarm in case of failures

Basic responsibilities – prompt offline processing After RAW is recorded to tape in CASTOR2 + Shuttle is done Processing is launched The processing is an automatic process Progress is displayed on the dashboard Beginning of data taking – automatic processing is disabled Lists of runs to be processed is compiled by the run coordinator / shift leader

Basic responsibilities – prompt offline processing (2) The experiment logbook contains ‘hints’ - run quality flags Per detector and global The run quality flags are presently filled manually, in the future by the Online QA Offline shifter responsibility is to follow for all PHYSICS runs the content of the quality flags and prompt the shift leader and the detector shifters to fill these ate EoR

Offline shifter check list Registration of RAW (dashboard) Periodic check of status Follow PHYSICS runs Ask shift leader in case of doubt Report registration errors to on-call expert The run copy and removal procedure – to be defined Shuttle (dashboard) Follow on processing of all runs + global Shuttle messages In case of preprocessor failures, escalate to (concerned) detector shifters In case of Shuttle failures first follow the restart/debug procedures, then report to on-call expert

Offline shifter check list (2) Fast reconstruction and event display (processing scripts on shifter console) Periodic check of PHYSICS runs (not the entire run!) Run reconstruction and analyse the AliRoot log files for errors/crashes Note the above in the shifter report pages and send to alice-shift-alarms@cern.ch Visualize periodically events in PHYSICS runs Note ‘strange’ event characteristics in the shifter report pages and send to alice-shift-alarms@cern.ch

Offline shifter check list (3) Data replication (dashboard) Periodic check of replication status Note ‘stuck’ runs – not replicated 12 hours after registration – in the shifter report pages and sent list to alice-shift-alarms@cern.ch Prompt data processing (dashboard) Periodic check of processing status Note ‘stuck’ runs – not processed 12 hours after registration – in the shifter report pages and sent list to alice-shift-alarms@cern.ch Shift report (shifter system) At end of shift – summary of the operation and noteworthy events

General shifter rules Before pressing the Read the procedures and rules, defined for each error type Try out the remedies If all fails, inform the on-call expert

Information sources for the shifter The shifter manual – instructions Was here Introducing the new Shifter interface Monitoring – MonALISA Dashboard Shuttle Processing and data management