The LHCb Online Framework for Global Operational Control and Experiment Protection F. Alessio, R. Jacobsson, CERN, Switzerland S. Schleich, TU Dortmund, Germany on behalf of the LHCb collaboration Conference on Computing in High Energy and Nuclear Physics Taipei, Taiwan, 19 October 2010

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN Motivations and Implications 2 LHC experiments have never in the past been so tightly connected to the accelerator, why? Enourmous stored energy - up to 2 x 360 MJ at nominal intensity and fragile experiments:  Protection of experiments  Monitor, understand, analyse and optimize experimental conditions Luminosity determination Automatic calibrations, global timing and beam/background monitoring Long-term detector stability and aging monitoring due to radiations  High density of operational communications and procedures  High level of reliability which requires direct communication interfaces at both HW and SW level  High level of interconnectivity, lot of redundancy High interaction rate and large events size:  Fast and reliable centralized readout, storage and transfer to offline processing  Fast feedback from Data Quality checking Many years of 24h operation with few people and non-experts:  Operating the whole detector from one console  Understandable high-level tools for diagnostics, alarms and data monitoring  Homogeneity and scalability of the system  Shifter training and gathering of information “at a glance”

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN LHCb Particularities 3 B Beam 2 from SPS (TI8) 1.Line of direct sight of the injection line High level of protection and reliability Beam stoppers which can produce high density showers Very complex background structure  Many parameters Full batch of up to 288*10^11 protons over 10  s = Energy of 2.4 MJ per single injection … 2.Fragile detector equipped with silicon instrumentation Vertex Locator, Silicon Trackers 3.Readout Electronics already ON at INJECTION, High Voltages OFF at INJECTION Switch ON/OFF coherently with LHC mode 4. Movable detector (VeLo), move IN only during PHYSICS 5. Global timing and readout control /event management centrally managed and relies on LHC parameters

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN 4 Large Interconnectivity 1.Protection through fast beam extraction 2.Automating and securing operational procedure 3.Readout Control and Event Management 4.Diagnostics, direct feedback, and real-time control for fast improvements 5.Analysis and feedbacks for optimization

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN LHCb Radiation Sources and Implications 5 Instantaneous damage Beam Interlock Background ………Trigger rates…………………… ……….Poor data quality………………… ……………………..…….Single event upsets………. …………..………Accelerated aging……….………… ………………….Long-term damage……………..... Online monitoring Accumulated dose and Luminosity Background Beam characteristics Machine settings Halo/beam-gas/………………....…….scraping……………....Beam incident Complex background structure requiring complete understanding and widely different level of reaction times Cover all ranges of losses and protect experiment Study fine time structure of losses Measure online beam characteristics and improve machine settings  reduce background and improve lumi/background

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN LHCb Defense & Safety 6 Inhibit injection or dump the beam 1.Vertex Locator out of garage position in “non-safe” operations 2.LHCb spectrometer magnet NOT OK 3.Diamond based Beam Conditions Monitor BCM dumped the beam (human analysis required) Master of Injection Inhibit CIBU (Beam Permit) CIBF (Injection Inhibit 1) CIBU (Injection Inhibit 2) “LHCb Detector Beam Control” BCM Read-Out & Injection Inhibit Interface (BCM TELL1) BCM Experiment Control System BIC VeLo Beam Interlock Magnet Status Interlock “Safe Beam Flags” for Mode dependence  Distributed via the General Machine Timing  Beam declared “safe” Handshakes  between machine and experiment to allow moving from a “safe”mode to an “unsafe” machine mode (INJECTION, ADJUST during physics and controlled DUMP state)

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN 7 LHCb Global Centralized Slow Operations VELO allowed IN Automation as function of LHC mode  Reduced 19 LHC states into 8 LHCb states by regrouping similar states  HV & LV of each sub-detector and data taking is controlled via LHCb state machine  New LHCb State is proposed and simply acknowledged by shifter (cross check)  Movable devices only allowed to move during the collision phase  Reliability and completeness to ensure to be in the right state at the right moment

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN Providing underlying framework for controlling HV, LV, VELO, DAQ and trigger in PVSS LHCb State Control Panel courtesy of Clara Gaspar

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN LHCb Global Real-Time Operations 9 Readout Supervisor LHC accelerator Beam Phase and Intensity Monitor Subdetectors Event Filter Farm L0 trigger RS Event Bank Events Requests Bunch currents Clock/orbit, UTC, LHC Parameters HW and run parameters Run statistics Luminosity Detector status L0 Decision RO Electronics Trigger Throttle Fast Readout Contrpol FE Electronics Interconnectivity in real-time information exchange for Readout Control and Event Management

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN Experimental Conditions – Online and Offline 10 Beam and background monitoring Timing monitoring Experimental conditions  Trigger rates  Online Luminosity  Bunch-by-bunch measurements  Beam gas  >10000 parameters LHC machine states LHC machine conditions  Bunch structure  Beam characteristics  > parameters Safety conditions SW Interlocks Magnet and radiation doses LHCb DataBases Processed data LHCb control room Alarms! Global centralized operations  HV/LV, LHCb states Timing correction Readout control LHC control room Webpages LHC operations  Tuning of machine LHCb Offline interactive analysis tools  Run Summaries, run “at a glance”  Experimental Analysis Tool (trends, plots, tables…)  Global operations webpage Shifters organization LHCb LHC Information exchange Servers LHC logging DataBases

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN 11 Complete HW and SW framework implemented  Beam Interlock System (BIS) pure hardware between user systems and LHC injection and beam dump  General Machine Timing (GMT) Slow machine timing and distribution of Safe Machine Parameters include Beam Flags  Beam Synchronous Timing (BST) Synchronicity to LHC beam instrumentation  Timing reception and monitoring (TTC) LHC clock and orbit distribution  Experiment-Machine Communication interfaces Gateways, technical network, services/commands  Software project within LHCb Experimental Control System  Post Mortem Telegrams (PMT) post mortem analysis of beam dump/accident  Software interlocks from LHCb particularities  Control of fill procedures, status of experiment and handshakes with machine  Beam and Background monitoring  Online Luminosity and online measurement of parameters/experimental conditions  Visual graph, alarms, Run Summary, Run Plan  Tools for quick offline analysis and web pages  LHCb Control room organization and shifter tools HW for high level of safety, high speed and ~100% reliability, imply a physical link SW for online monitoring, offline analysis, alarms, human-PC interactions, experiment- machine communication, slow controls Conclusion

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN 12 Complete HW and SW framework implemented  Constantly used, stable, high success rate  Heavy contribution to machine commissioning for the entire running period thanks to archiving and interactive tool  Continuously used for LHCb optimization  Ready in time: experiments are starting to be sensitive not only to luminosity, but also to beam currents!! Conclusion

CHEP2010, Taipei, Taiwan, 19/10/10 F. Alessio, CERN Backup 13