LHCb Computing activities Philippe Charpentier CERN – LHCb On behalf of the LHCb Computing Group.

Slides:



Advertisements
Similar presentations
31/03/00 CMS(UK)Glenn Patrick What is the CMS(UK) Data Model? Assume that CMS software is available at every UK institute connected by some infrastructure.
Advertisements

Buffers & Spoolers J L Martin Think about it… All I/O is relatively slow. For most of us, input by typing is painfully slow. From the CPUs point.
Clara Gaspar on behalf of the LHCb Collaboration, “Physics at the LHC and Beyond”, Quy Nhon, Vietnam, August 2014 Challenges and lessons learnt LHCb Operations.
Stripping Plans for 2014 and 2015 Laurence Carson (Edinburgh), Stefano Perazzini (Bologna) 2 nd LHCb Computing Workshop,
Large scale data flow in local and GRID environment V.Kolosov, I.Korolko, S.Makarychev ITEP Moscow.
Notes on offline data handling M. Moulson Frascati, 29 March 2006.
LHCb Quarterly Report October Core Software (Gaudi) m Stable version was ready for 2008 data taking o Gaudi based on latest LCG 55a o Applications.
December 17th 2008RAL PPD Computing Christmas Lectures 11 ATLAS Distributed Computing Stephen Burke RAL.
Computing and LHCb Raja Nandakumar. The LHCb experiment  Universe is made of matter  Still not clear why  Andrei Sakharov’s theory of cp-violation.
D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina, L.Lueking,
Computing Infrastructure Status. LHCb Computing Status LHCb LHCC mini-review, February The LHCb Computing Model: a reminder m Simulation is using.
LHCb computing in Russia Ivan Korolko (ITEP Moscow) Russia-CERN JWGC, October 2005.
Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.
9 February 2000CHEP2000 Paper 3681 CDF Data Handling: Resource Management and Tests E.Buckley-Geer, S.Lammel, F.Ratnikov, T.Watts Hardware and Resources.
1 LCG-France sites contribution to the LHC activities in 2007 A.Tsaregorodtsev, CPPM, Marseille 14 January 2008, LCG-France Direction.
Meeting, 5/12/06 CMS T1/T2 Estimates à CMS perspective: n Part of a wider process of resource estimation n Top-down Computing.
ROOT and Federated Data Stores What Features We Would Like Fons Rademakers CERN CC-IN2P3, Nov, 2011, Lyon, France.
Update on replica management
LHCb The LHCb Data Management System Philippe Charpentier CERN On behalf of the LHCb Collaboration.
1 LHCb on the Grid Raja Nandakumar (with contributions from Greig Cowan) ‏ GridPP21 3 rd September 2008.
Simulation Commissioning, Validation, Data Quality A brain dump to prompt discussion Many points applicable to any of LHCb software but some simulation.
Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.
LHCbComputing LHCC status report. Operations June 2014 to September m Running jobs by activity o Montecarlo simulation continues as main activity.
LHCbComputing Resources requests : changes since LHCb-PUB (March 2013) m Assume no further reprocessing of Run I data o (In.
20 Dec Peter Clarke Computing production issues 1.Disk Space – update from high-mu workshop 2.Disk planning 3.Use of T2s NCB meeting 20-Dec-2010.
WLCG LHCC mini-review LHCb Summary. Outline m Activities in 2008: summary m Status of DIRAC m Activities in 2009: outlook m Resources in PhC2.
Large scale data flow in local and GRID environment Viktor Kolosov (ITEP Moscow) Ivan Korolko (ITEP Moscow)
LHCb report to LHCC and C-RSG Philippe Charpentier CERN on behalf of LHCb.
David Stickland CMS Core Software and Computing
Computing Issues for the ATLAS SWT2. What is SWT2? SWT2 is the U.S. ATLAS Southwestern Tier 2 Consortium UTA is lead institution, along with University.
Workflows and Data Management. Workflow and DM Run3 and after: conditions m LHCb major upgrade is for Run3 (2020 horizon)! o Luminosity x 5 ( )
LHCb Readiness for Run WLCG Workshop Okinawa
OPERATIONS REPORT JUNE – SEPTEMBER 2015 Stefan Roiser CERN.
ATLAS Distributed Computing perspectives for Run-2 Simone Campana CERN-IT/SDC on behalf of ADC.
Handling of T1D0 in CCRC’08 Tier-0 data handling Tier-1 data handling Experiment data handling Reprocessing Recalling files from tape Tier-0 data handling,
1 LHCb computing for the analysis : a naive user point of view Workshop analyse cc-in2p3 17 avril 2008 Marie-Hélène Schune, LAL-Orsay for LHCb-France Framework,
LHCbDirac and Core Software. LHCbDirac and Core SW Core Software workshop, PhC2 Running Gaudi Applications on the Grid m Application deployment o CVMFS.
LHCb status and plans Ph.Charpentier CERN. LHCb status and plans WLCG Workshop 1-2 Sept 2007, Victoria, BC 2 Ph.C. Status of DC06  Reminder:  Two-fold.
D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, R. Brock,T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina,
Markus Frank (CERN) & Albert Puig (UB).  An opportunity (Motivation)  Adopted approach  Implementation specifics  Status  Conclusions 2.
Computing Model José M. Hernández CIEMAT, Madrid On behalf of the CMS Collaboration XV International Conference on Computing in High Energy and Nuclear.
LHCb 2009-Q4 report Q4 report LHCb 2009-Q4 report, PhC2 Activities in 2009-Q4 m Core Software o Stable versions of Gaudi and LCG-AA m Applications.
Jianming Qian, UM/DØ Software & Computing Where we are now Where we want to go Overview Director’s Review, June 5, 2002.
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
Status Report on Data Reconstruction May 2002 C.Bloise Results of the study of the reconstructed events in year 2001 Data Reprocessing in Y2002 DST production.
CMS data access Artem Trunov. CMS site roles Tier0 –Initial reconstruction –Archive RAW + REC from first reconstruction –Analysis, detector studies, etc.
LHCbComputing LHCb computing model in Run1 & Run2 Concezio Bozzi Bologna, Feb 19 th 2015.
LHCb LHCb GRID SOLUTION TM Recent and planned changes to the LHCb computing model Marco Cattaneo, Philippe Charpentier, Peter Clarke, Stefan Roiser.
LHCb Computing 2015 Q3 Report Stefan Roiser LHCC Referees Meeting 1 December 2015.
15.June 2004Bernd Panzer-Steindel, CERN/IT1 CERN Mass Storage Issues.
Using volunteered resources for data-intensive computing and storage David Anderson Space Sciences Lab UC Berkeley 10 April 2012.
ATLAS – statements of interest (1) A degree of hierarchy between the different computing facilities, with distinct roles at each level –Event filter Online.
1 LCG-France 22 November 2010 Tier2s connectivity requirements 22 Novembre 2010 S. Jézéquel (LAPP-ATLAS)
THE ATLAS COMPUTING MODEL Sahal Yacoob UKZN On behalf of the ATLAS collaboration.
LHCb distributed computing during the LHC Runs 1,2 and 3
L’analisi in LHCb Angelo Carbone INFN Bologna
Data Challenge with the Grid in ATLAS
The LHCb Software and Computing NSS/IEEE workshop Ph. Charpentier, CERN B00le.
Bernd Panzer-Steindel, CERN/IT
Status and Prospects of The LHC Experiments Computing
LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.
Philippe Charpentier CERN – LHCb On behalf of the LHCb Computing Group
Simulation use cases for T2 in ALICE
ALICE Computing Upgrade Predrag Buncic
R. Graciani for LHCb Mumbay, Feb 2006
LHCb Computing Philippe Charpentier CERN
LHCb status and plans Ph.Charpentier CERN.
ATLAS DC2 & Continuous production
The ATLAS Computing Model
The LHCb Computing Data Challenge DC06
Presentation transcript:

LHCb Computing activities Philippe Charpentier CERN – LHCb On behalf of the LHCb Computing Group

LHCb Computing activities PhC2 Summary of Computing activities m Simulation o Mainly used for identifying background and evaluating acceptances and efficiencies o Simulates an ideal detector, however with realistic geometry o Event generation and detector response tuned to real data P Iterative process, depends on the data taking yesr m Real data handling and processing o Distribution to Tier1s (RAW) o Reconstruction (SDST) o Stripping and streaming (DST) o Group-level production (µDST) m User analysis o MC and real data processing o Detector and efficiency calibration o End-user analysis (usually off-Grid: Tier3 or desktop)

LHCb Computing activities Simulation jobs m 5 steps jobs m Gauss: simulation, based on Geant4 m Boole: digitisation m Moore: trigger m Brunel: reconstruction m DaVinci: stripping (single stream) m Any file may be saved, usually only the final DST (currently uploaded to CERN) m 100 to 200 events per job m 5 to 10 hours duration m 40 to 80 MB per file m Merging required (see later) PhC3 SIM DIGI DST (X)GEN Gauss Boole Brunel DaVinci DIGI Moore

LHCb Computing activities Reconstruction-Stripping jobs m 2 steps jobs m One input file: copy to local disk m Brunel: reconstruction m DaVinci: stripping and streaming o Around 10 to 13 streams m SDST and DSTs saved (locally) o SDST: local T1D0 (LHCb_RDST) o (µ)DSTs: temporary T0D1 (LHCb_DST) m RAW files up to 3 GB (45,000 events) m 2s per event: 1 day jobs m SDST: 80% of RAW size m All DSTs: ~10% of RAW size o Individual files small (10’s MB) PhC4 SDST DST Brunel DaVinci DST µDST RAW

LHCb Computing activities Stripping jobs m One step jobs m Multiple input files (e.g. 10 SDSTs) o RAW files required as well o All files must be present on disk cache (possibly staged) o Access by protocol (xroot, dcap…) m DSTs 10 times bigger than for Reco-Stripping o Saved locally (LHCb_DST) o Streamed DSTs ~ 100’s MB o Merging still required PhC5 SDST DST DaVinci DST µDST RAW

LHCb Computing activities Merging jobs m Automatically generated from job output o Simulation o Reco-stripping or Stripping m Typically 5 GB files o For real data, only merge data from the same run o Run duration can easily be adjusted online P was 1 hour in 2010 (7 Mevts, ~200 RAW files of 3 GB) m Uses download policy for input data o Limited by number of files: P SRM overload P SE and disk overload o See Ricardo’s talk tomorrow m Merged files uploaded locally to T1D1 (master copy) o LHCb_M-DST or LHCb_MC_M-DST PhC6

LHCb Computing activities Data replication m Performed by a data driven transformation o Same mechanism as productions P See Andrew Smith’s talk at CHEP10 m Distribution policy implements the LHCb Computing Model o RAW files: one Tier1 from the Tier0 replica o MC: 2 master replicas (CERN + Tier1), 1 secondary replica P Sites selected randomly P Foresee to implement space driven policy o Real data: 2 master replicas (CERN + Tier1), 2 secondary replicas P Differs from CM (should be 5 secondary replicas) P Adaptation following larger event sizes P Each run is distributed to the same sites m Replication using FTS PhC7

LHCb Computing activities RAW streams 2010 m In 2010, two RAW streams: o EXPRESS: selected small rate (typically 5 Hz) P Used for Data Quality checking P Smaller files (200 MB), kept at CERN only P CERN, quasi online (less than 2 hours) P DQ result used to allow FULL processing o FULL: all triggers, around 2,000 Hz P Distributed to Tier1s (after successful tape migration) P Using the CPU pledge share for distribution d Share set to 0 when site is unavailable P Full runs assigned to single site ( files) P Reconstruction-Stripping launched automatically after the run is declared OK P Merging performed on the same site PhC8

LHCb Computing activities RAW streams 2011 m 2011 (options): o EXPRESS as 2010 o MINBIAS: around 200 Hz of minimum/no bias triggers P Special stream, treated like the FULL stream o b-physics triggers and charm-physics. Options: P 3,000 Hz on FULL stream P 2,000 Hz on BPHYS stream, 1,000 Hz on CHARMPHYS stream P To be decided soon P Replication to Tier1s as for FULL P Staged processing for CHARMPHYS triggers (depending on CPU availability) with lower priority m Processing o Run Reco and Stripping in separate jobs o Alleviates the problem of large number of files to be merged P Use SDSTs as input to stripping jobs P Caveat: use protocol access, SRM possibly overloaded at start of jobs (get tURLs at once) PhC9

LHCb Computing activities Additional possible changes m Reconstruction at Tier2: o Using files download from a well connected Tier1 o Selected number of Tier2s o Better control than Analysis P All handled by the Production team, better organised, less chaotic o Most probably needed for end-of-year reprocessing m Analysis at Tier2s: o Still to be commissioned P Do we need it? o Problem of local manpower and storage management m LHCb prefers reconstruction to analysis o For the time being, no limitation seen for user jobs at Tier1s P Turn-around still very good (all user jobs finished overnight) PhC10