1. Univ. Milano-Bicocca e INFN
Data challenge CMS
M. Paganoni
Univ. Milano-Bicocca e INFN

2. Contributors
CNAF: William Bacchi, Daniele Bonacorsi, Paolo Capiluppi, Giuseppe Codispoti, Alessandra Fanfani
T2 Roma: Luciano Barone, Pietro Govoni, Martina Malberti, Paolo Meridiani, Giovanni Organtini, Shahram Rahatlou, Francesco Safai Tehrani
T2 LNL: Massimo Biasotto, Federica Fanzago, Ugo Gasparini, Martino.Margoni, Gaetano Maron, Ezio Torassa
Pisa: Filippo Ambroglini, Giuseppe Bagliesi, Paolo.Bartalini, Tommaso Boccali, Federico Calzolari, Livio Fanò
Bari: Nicola De Filippis, Giacinto Donvito, Giorgio Maggi

3. Computing Software Analysis Challenge 2006
A 50 million event exercise to test the workflow and dataflow as defined in the CMS computing model
A test at 25% of the capacity needed in 2008
Main components:
Preparation of large MC simulated datasets (some with HLT-tags)
Prompt reconstruction at Tier-0:
Reconstruction at 40 Hz (over 150 Hz) using CMSSW
Application of calibration constants from offline DB
Generation of Reco, AOD, and AlCaReco datasets
Splitting of an HLT-tagged sample into 10 streams
Distribution of all AOD & some FEVT to all participating Tier-1s
Calibration jobs on AlCaReco datasets at some Tier-1s and CAF
Re-reconstruction performed at Tier-1s
Skim jobs at some Tier-1s with data propagated to Tier-2s
Physics jobs at Tier-2s and Tier-1s on AOD and Reco
Italian contribution

4. Official Timeline
– June 1: computing systems ready for Service Challenge SC4
– June 15: physics simulation validation complete
– July 1: start MC production
– Aug.15: calibration, alignment, HLT, reconstruction, and analysis tools ready
– Aug.30: 50 Mevt produced, 5M with HLT pre-processing
– Sep. 1: Computing systems ready for CSA
– Sep 15: Start CSA06
– Oct 1: start smooth operation for CSA06
– Oct 30: End smooth operation for CSA06
– Nov 15: Finish CSA06

5. Success metrics Item Goal Threshold Result # Tier1 7 5 # Tier2 20 1524
Weeks of sustained rate 4 2
Tier0 efficiency
80 %
30 %
100 %
Running jobs per day (2h) Tier1+Tier2
50k
30k
Grid job efficiency
90 %
75 %
95 %
Data serving (storage to CPU)
1 MB/s/slot
300 MB/s (T1) 100 MB/s (T2) OK
Data transfer Tier0-all Tier1 (tape)
150 MB/s
75 MB/s
550 MB/s
Data transfer Tier1 - Tier2
20 MB/s
5 MB/s

6. Computing resources Tier-0 (CERN): 1.4M SI2K (~ 1400 CPUs at CERN)
240 TB
Tier-1 (7 sites):
2500 CPUs in total
70 TB disk + tape as minimum to participate
Tier-2 (25 sites):
2400 CPUs in total
Average 10 TB disk at participating Tier-2

7. MC production software and tools
ProdAgent tool used to automatise the production:
consists of many agents running in parallel: JobCreator, JobSubmitter, JobTracking, MergeSensor….
ouput files are registered in Data bookkeping service (DBS); blocks of files are registered in Data Location System (DLS); local catalogue used to map LFNs to local PFNs
Files are merged for optimum size before transfer to CERN
CMSSW installed via grid tools or directly by site admins in remote sites.
Storage management deployed: CASTOR, dCache, DPM

8. MC pre-production 4 production teams active:
1 for OSG
-- Ajit Mohapatra – Wisconsin
(taking care of 7 OSG CMS Tier2)
3 for LCG:
-- Jose Hernandez – Madrid (Spain, France, Belgium, CERN)
-- Carsten Hof – Aachen (Germany, Estonia, Taiwan, Russia,
Switzerland, FNAL)
-- Nicola De Filippis – Bari (Italy, UK, Hungary)
Large partecipation of CMS T1s and T2s involved

9. Monitoring of production
via web interface
First prototype of monitoring was developed by Bari team:

10. Monitoring of MinBias (1)
Maximum rate per day: 1.15 M

11. Monitoring of MinBias (2)
CNAF
LNL
Roma
Bari
Pisa
Most of the failures at CNAF were related to stageout and stagein problems with CASTOR2

12. 12 M of events produced by the LCG(3) team
Dataset statistics
Total: ~ 66 M events No Pileup Total FEVT: O(150) TB
1. Minimum bias (40M)
2. Zµµ (2M)
3. We (4M)
4. t-tbar (6M) [all decays]
5. Electroweak soup (5M)
Wl nu + Drell-Yan (m>15 GeV) + WW + HWW
6. HLT soup (5M): 10 effective MC HLT triggers
W (leptons) + Drell-Yan (leptons) + t-tbar (all modes) + dijets
7. Jet calibration soup (1M)
dijet + Z+jet
8. Soft Muon Soup (2M)
Inclusive muons in minbias + J/Psi production
9. Exotics Soup (1M)
LM1 SUSY, Z’ (700 GeV), and excited quark (2000 GeV) [all decays]
for calibration
12 M of events produced by the LCG(3) team

13. Efficiency and problems
Overall efficiency: 88%
Probability for a job to end successfully once it is submitted
Grid efficiency: 95%
Aborted jobs: jobs not submitted because requirements not met (merge jobs) or jobs once submitted fail due to Grid infrastructure reason
Problems:
stage out was the main cause of job failures. More robust checking were implemented, more attempts to stage, a fallback strategy etc..
merge jobs caused tipically an overload of the storage system because of the high rate of read access; CASTOR2 at CNAF was tuned to cope with the needs of the production (D. Bonacorsi and CNAF admins)
site validation: storage, software tag, software mount points, matching of CE
consistency between fileblock/files in DBS/DLS and the reality at sites.
Support of Italian Tier-1 and Tier-2 very effective also in August

14. Tier-0 tasks in CSA06 Reconstruction with CMSSW_1_0_x (x6)
All main reconstruction components included
Detector-specific local reconstruction and clustering
Tracking (only 1 algo used), vertexing, standalone , jets
Global  (with tracker), electrons, photons, b&tau tagging
Reconstruction time small: 4.5s/ev MB, 20s/ev ttbar
Computing model assumes 25 s/ev
Calibration/Alignment
Ability to pull in constants from Offline DB included for ECAL, Tracker, and Muon reconstruction
Direct access to Oracle or via Frontier cache

15. Tier-0 operations 4 weeks uptime (goal), 207M events processed
2 Oct. : operations at Tier0 started
First week mostly minbias (with some EWK) using CMSSW102 while bugs fixed to improve robustness on signal samples
Second week processing included signal samples at rates generally matched to T1 bandwidth metrics and using CMSSW103
After having run for about 23 days, 120M events at 100% uptime, decided to increase scale for last days
Reprocessed all signal samples in ~5 days using CMSSW106 and maximum CPU usage
Performance:
160 Hz processing rate, peaking at 300 Hz
1250 CPUs for prompt reconstruction
150 CPUs for AOD and AlCaReco production (separate step)
All constants pulled from Frontier
i.e. full complexity of CSA exercise
4 weeks uptime (goal), 207M events processed

16. Calibration/Alignment exercise at Tier-0 CAF
Calibration/alignment tasks:
Specialized tasks to align/calibrate subsystems using start-up miscalibrated samples, e.g.
Align a portion of Tracker with HIP algorithm by using Z →mm sample on the central analysis facility (CAF) for prompt calibration/alignment
Intercalibrate ECAL crystals by phi symmetry in minbias events, 0/, or by isolated electrons from W/Z
Specialized reduced RECO data format (AlCaReco) to be used for calibration/alignment stream from Tier-0
Mechanism to write constants back into offline DB to be used
Re-reconstruction at Tier-1 required to test new constants
Propose that miscalibration is applied at RECO

17. Tracker Alignment exercise
CSA06 misalignment scenario: TIB dets and TOB rods misaligned by applying:
random shifts, drawing from a flat distribution of witdth +/-100 mm, in (x,y,z) for the double sided modules and in x (sensitive coordinate) for the single sided ones
random rotations, drawing from a flat distribution of witdth +/-10 mrad, in (alpha,beta,gamma) for all the modules
TIB double sided dets positions
Alignment exercise:
to read the object in the DB, to apply the initial misalignment;
to run the iterative HIP algorithm and to determine alignment constants;
1M events used and 10 iterations.
jobs running in parallel on 20 CPUs on a dedicated queue at Tier-0;
new costants inserted into the DB

18. Transfer Tier-0/Tier-1s
All 7 Tier-1 participated in the challenge performing very well
some storage element software or hardware problems at individual sites
The longest down time at any site has been about 18 hours
Files are injected into the CMS data transfer system PhEDEx and transferred using FTS
Highest rate from CERN was 550MB/s
First 3 Week Average
Site
Rate
ASGC
14.3MB/s
CNAF
18.0MB/s
FNAL
47.8MB/s
GridKa
21.7MB/s
IN2P3
14.6MB/s
PIC
14.4MB/s
RAL
16.4MB/s
Total
147MB/s

19. Transfer Tier-0/Tier-1s

20. Skimming data at Tier-1s
To fit data at T2, and to reduce primary datasets to manageable sizes, it was needed to run skim jobs at T1s to select events according to the analyses
Skim configuration files prepared according to the RECO and AOD format (also including some “MC truth” information)
Organized skim jobs ran with ProdAgent
Different skim procedures prepared by the users for running on the same dataset were unified in a single skim job producing different streams
10 filters prepared by the Italian people to cope with the analyses prepared
4 teams for running skim jobs at tier-1s
N. De Filippis: Electroweak soup (RAL, CNAF, ASGC, IN2P3)
D. Mason: Jets (FNAL)
C. Hof: TTbar ( FZK and FNAL)
J. Hernandez: Zmumu (PIC and CNAF)
Skim job output files shipped to Tier-2s for end-user analyses
9 Oct. – T1 Skim jobs started

21. Monitoring of skim
jobs at Tier-1s

22. Transfer of skim outputs from Tier-1s to Tier-2s
Problems related to:
wrong config. of Tier-2 sites
wrong setup of download agents with FTS
CNAF related problems (FTS server, CASTOR)

23. Total transfer Tier-0 to
Tier-1s and Tier-2s
Exceeded 1PB in 1 month!

24. Analyses at Tier-2s (1)

25. minimum bias underlying event
Analyses at Tier-2s (2)
All INFN Tier2s took part to the last step of the CSA06: the physics analyses starting from the output of skim procedures
LNL
Wmn selection
Pisa
(tau validation)
minimum bias underlying event
Rome
electron reco
Bari
tracker misalignment

26. Analysis at Rome Three analyses with goals:
to study of the electron reconstruction in Z  ee events (Meridiani)
to measure the W mass in W  en events (Tabarelli, Malberti)
to run a simple calibration with W  en events (Govoni)
Electron and Z mass reconstruction using the hybrid supercluster
Eff vs pT
Eff vs h mZ

27. Analysis at Pisa (1)
The general idea is to simulate a "early data taking" activity of the t group:
study the tau tag efficiency from the Z tt events
study the misidentification with the recoiling jet with Z+jet, Z  mm events
In addition: run t validation package on skimmed events pure di-tau sample and ttbar sample (S. Gennai, G. Bagliesi).
Isolation efficiency vs Isolation Cone :
pT of the jet

28. Analysis at Pisa (2)
Study of minimum bias/underlying event (Fanò, Ambroglini, Bartalini)
Monte Carlo tuning for LHC
Pileup undestanding
UE contribution measurements in MB events UE
MinBias

29. Analysis at LNL
Goal: to study the W mn preselection with different Monte Carlo data samples
Two data samples were considered (Torassa, Margoni, Gasparini):
(1) the electroweak soup (3.4 M evts, 50% Wmn and 50% DY)
(2) the soft muons (1.8 M evts, 50% minimum bias and 50% J/y, pTm > 4 GeV)
EWK soup
GlobalMuon reconstructor

30. Analysis at Bari
Goals: to study the effect of tracker misalignment on track reconstruction performances (De Filippis):
with the perfect tracker geometry;
in the short term and in the long term misalignment scenario by reading misalignment position and errors via frontier/squid from the offline database ORCAOFF.
by using the tracker module position and errors as obtained by the output of the alignment process that will be run at CERN T0.
Data samples used: Z→mm and ttbar (the second for computing the fake rate)

31. Analysis jobs at Bari grid efficiency = 99 %, appl. eff = 94 %
CRAB_1_4_0 used to submit 1.8 k jobs
grid efficiency = 99 %, appl. eff = 94 %
Bunch of 150 jobs run in different time slots
max 45 jobs run in parallel
the configuration of squid tuned to ensure that the alignment data were read by the local cache of squid via the frontier client rather than from CERN (blue histo).
 frontier/squid works as expected at tier-2 Bari when accessing alignment data

32. Re-reconstruction at Tier-1s
The last step of CSA06:
Re-reconstruction at Tier-1s
Goals:
to demonstrate re-reconstruction from some RAW data at Tier-1s as part of the calibration exercise
Status:
access of Offline database via frontier working
re-reconstruction demonstrated at ASGC, FNAL, IN2P3, PIC and CNAF
Running at RAL and further tests at CNAF
PIC

33. Tier2 - LNL Import: 200TB transferred, rate 20-50 MB/s
Export: 60TB transferred, rate 5-20 MB/s
SC4 challenge (Jun-Sep 06)
fake analysis jobs
MC production
user analysis
Jun-Sep 2006 (~50K jobs)

34. Tier2 - LNL (2) Total number of jobs running (last 6 months)
CMS CSA06
CMS MC production

35. Tier2 - Roma
Nuove cpu installate 12/10/06

36. Pisa
/day
fake analysis jobs
MC production

37. Bari

38. Conclusions
CSA06 was supposed to be a challenge to commission the computing/software/analysis system but in some cases it required also development/deployment of the tools
CSA06 was successful at INFN (all the steps were executed) but thanks to the 100 % work of few experts and to the coordinated effort of many people at Tier-1 and Tier-2 sites.
CSA06 analysis exercises need to be he ramp-up for the physics program/organization in Italy
CSA07 should cope with both simulated and real data and focus on start-up operations (calibration and alignment) and analysis preparation

39. Rimodulazione

40. Richieste 09/2006 1 TB = 1.5 kEuro 1 box = 8 kSI = 3 kEuro
Le richieste totali (inclusi SJ) porterebbero sul totale dei 4 centri di calcolo ad avere un fattore 2 sotto il piano iniziale, per tenere conto della rimodulazione di LHC (248 TB contro 530 TB e 1014 kSI2K contro 1600 kSI2K)