1. LHCb report to LHCC and C-RSG Philippe Charpentier CERN on behalf of LHCb

2. LHCb to LHCC and C-RSG review, PhC2 Activities in 2009-Q3/Q4 m Core Software o Stable versions of Gaudi and LCG-AA m Applications o Stable as of September for real data o Fast minor releases to cope with reality of life… m Monte-Carlo o Intensive MC09 simulation (@ 5TeV) P Minimum bias P b- and c- inclusive P b signal channels o Few events in foreseen 2009 configuration (450 GeV) o MC09 stripping (2 passes) P Trigger stripping P Physics stripping m Real data reconstruction and stripping o As of November 20 th …

3. Resource usage LHCb to LHCC and C-RSG review, PhC3

4. 139 sites hit, 4.2 million jobs m Start in June: start of MC09 LHCb to LHCC and C-RSG review, PhC4

5. Job failure: 15% (17% at Tier1s) LHCb to LHCC and C-RSG review, PhC5

6. Failure breakdown LHCb to LHCC and C-RSG review, PhC6

7. Production and user jobs LHCb to LHCC and C-RSG review, PhC7

8. Jobs at Tier1s LHCb to LHCC and C-RSG review, PhC8

9. Job types at Tier1s LHCb to LHCC and C-RSG review, PhC9

10. CPU used (not normalised) LHCb to LHCC and C-RSG review, PhC10 m Average job duration o 5.6 hours for all jobs o 20 mn for user jobs (20%) o 6.6 hours for production jobs

11. m Average job duration o 5.6 hours for all jobs o 20 mn for user jobs o 6.6 hours for production jobs LHCb to LHCC and C-RSG review, PhC11

12. CPU usage (not normalised) LHCb to LHCC and C-RSG review, PhC12

13. WLCG vs LHCb accounting (unnormalised) m 13% more in WLCG than in DIRAC (unnormalised) o 1.26 Mdays vs 1.1 Mdays o Overhead of non reporting jobs + pilot/LCG/batch frameworks m Average CPU power: 1.5 kSI2k (from WLCG accounting) LHCb to LHCC and C-RSG review, PhC13

14. Normalised CPU usage in 2009 m Ramping up of pilot role in summer m Resource usage decreased since LHC restarted o Concentrate on (few) real data o Wait for data analysis for continuing MC simulation LHCb to LHCC and C-RSG review, PhC14 m Group 1: production m Group 2: pilot m Group 3 & 4: user m Group 5: lcgadmin

15. Resource usage LHCb to LHCC and C-RSG review, PhC15 m Note: CERN above does not include non-Grid usage o From WLCG accounting: 32% is non-Grid at CERN o CERN number should then read: 2.18 kHS06.years m CPU usage within 10% of requests m Distribution not exactly like expected o More non-Tier1 resources available P Less MC ran at CERN + Tier1s o Almost no real data: less resources used at CERN P CAF not used as much as expected SiteUsed (kHS06.years)Requested (kHS06.years) CERN1.488.54 Tier1s8.2411.7 Tier2s24.4417.12 Total34.1637.36

16. Storage usage LHCb to LHCC and C-RSG review, PhC16 m *) From Castor queries today m **) From WLCG accounting end December m ***) Including 420 TB for T1D0 cache m Sites provided slightly more than the pledges o Thanks! o At CERN, some disk pools (default, T1D0) were not included in the requests but are in the accounting SiteRequestedAllocatedUsed CERN *) TxD1650696.5482.7 CERN *) T1D070148.5 irrelevant CERN **) 720721478 Tier1s **) 1740 ***) 1915633

17. Experience with real data LHCb to LHCC and C-RSG review, PhC17

18. First experience with real data m Very low crossing rate o Maximum 8 bunches colliding (88 kHz crossing) o Very low luminosity o Minimum bias trigger rate: from 0.1 to 10 Hz o Data taken with single beam and with collisions LHCb to LHCC and C-RSG review, PhC18 No zero-suppression in VELO Otherwise ~25 GB only!

19. Real data processing m Iterative process o Small changes in reconstruction application o Improved alignment o In total 7 sets of processing conditions P Only last files were all processed 4 times now (twice in 2010) m Processing submission o Automatic job creation and submission after: P File is successfully migrated in Castor P File is successfully replicated at Tier1 o If job fails for a reason other than application crash P The file is reset as “to be processed” P New job is created / submitted (automatic) o Processing more efficient at CERN (see later) P Eventually after few trials at Tier1, the file is processed at CERN o No stripping ;-) P DST files distributed to all Tier1s for analysis LHCb to LHCC and C-RSG review, PhC19

20. Reconstruction jobs LHCb to LHCC and C-RSG review, PhC20

21. Issues with real data m Castor migration o Very low rate: had to change the migration algorithm for more frequent migration (1 hour instead of 8 hours) m Issue with large files (above 2 GB) o Real data files are not ROOT files but open by ROOT o There was an issue with a compatibility library for slc4-32 bit on slc5 nodes P Fixed within a day m Wrong magnetic field sign o Due to different coordinate systems for LHCb and LHC ;-) o Fixed within hours m Data access problem (by protocol, directly from server) o Still dCache issue at IN2P3 and NIKHEF P dCache experts working on it o Moved to copy mode paradigm for reconstruction o Still a problem for user jobs: a pain! P Sites are regularly banned for analysis LHCb to LHCC and C-RSG review, PhC21

22. Transfers and job latency m No problem observed during file transfers o Files randomly distributed to Tier1 o Will move to distribution by runs (few 100’s files) o For 2009, runs were never longer than 4-5 files! o Max file size set to 3 GB m Very good Grid latency o Time between submission and jobs starting running LHCb to LHCC and C-RSG review, PhC22

23. Resource requests LHCb to LHCC and C-RSG review, PhC23

24. Resource requests for 2010-12 m 2010 running o The requests were made in April-June 2009 P No additional resources expected P Try to fit within those requests o Running scenario for LHCb P March: 35% LHC efficiency @ 100 Hz P April-May-June: 50% LHC efficiency @ 1 kHz in average P July-August-September-half October: 50% @ 2 kHz P no Heavy Ion run for LHCb P This corresponds to 6.1 10 6 seconds @ 2 kHz P The 2009-10 request accounted precisely by chance for 6.1 10 6 seconds (0.5+5.6) P Therefore we use 6.1 10 6 seconds for 2010 at 2 kHz trigger rate m 2011 running o Use the recommendation of MB P March: 35% LHC efficiency @ 2 kHz P April to mid-October: 50% LHC efficiency @ 2 kHz P Total running time: 8.9 10 6 seconds m 2012: no run LHCb to LHCC and C-RSG review, PhC24

25. Resource requirements for 2010-12 LHCb to LHCC and C-RSG review, PhC25 kHEP06*year 2010 (old)2010 (confirmed)2011 (prelim.)2012 (very prelim.) Integrated PowerIntegratedPowerIntegratedPower CERN T05.704.504.07 CERN CAF - Analysis/Calib/Alignm ent 11.5611.9115.46 CERN T0 + T117.1917.262116.412019.5324 Tier1s32.9933.844157.497065.5580 Tier2s31.74 4631.484631.4846 Total81.9182.83108105.38136116.57150 Disk (TB) CERN T0 + T11290127016851776 Tier1s3290335042154458 Tier2s20 Total4600464059206254 Tape (TB) CERN T0 + T11500146230203723 Tier1s1800192242715605 Total3300338472909328

26. Comments on resources m Very uncertain and fluctuating running plans! m Depending on LHC running, MC requests may be different o Minimum bias, charm physics, b physics… m Only after one year (at least) experience we can see how running analysis on the Grid works o Analysis at CERN? o Analysis at Tier3s? o Reliability for analysis? m 2012 is still very uncertain o No LHC running o Will the MC requests be the same as previous years o How many reprocessings? P Currently assume 1 full reprocessing of 2010 and 2 of 2011 LHCb to LHCC and C-RSG review, PhC26

27. Conclusions m Real data in 2009 o So few that it didn’t impact resource usage o Was extremely valuable for P Setting procedures P Start understanding the detector d Already very promising performance after a few days  Π 0 peak, Λ and K 0 reconstruction… P Exercising automatic processes m 2010 o Still expect somewhat chaotic running P Frequent changes in LHC settings, LHCb trigger commissioning o No change in LHCb resource requests w.r.t. June 2009 m 2011 o More precise requests with experience from 2010 m 2012 o Still very preliminary, but small increase only compared to 2011 LHCb to LHCC and C-RSG review, PhC27