1. 1 RDMS CMS Computing: Russian CMS Tier-2 and Tier-1 for CMS at JINR llya Gorbunov and Sergei Shmatov Joint Institute for Nuclear Research, Dubna shmatov@cern.ch School on JINR/CERN GRID and Advanced Information Systems “ Joint Institute for Nuclear Research, Dubna May 14-18, 2012

2. 2  Compact Muon Solenoid goals and performance participation of Russian and Dubna Member State Institutions data taking, triggering and selection data flows  CMS Computing Model data model data flows data processing Tier-1/Tier-2 requirements and resources  RDMS Computing Model RDMS CMS Tier-2s (readiness, job status, data transfers) computing for remote operations (JINR ROC) CMS Tier-1 at Dubna (plans and prospects) Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

3. Brief Intro: What does physics require from Computing? 3 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

4. Why we need more accelerators and experiments ? Being the remarkable event on the long way of searching of the true, Standard Model of interactions is powerless to answer on very important questions: Origin of masses of fundamental blocks (quarks, leptons) A reason for three generations of matter fermions A reason of domination matter over antimatter and formation of observable Universe structure A source of dark mater in the Universe Possibility of high energy unification of three forces We hope we will be able to answer on these questions at least partially with next generations of accelerators And after all, what about gravity? 4 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

5. Open Questions:  Mass: why do fundamental particle have mass and why these masses are such different?  Antimatter: why at the present days matter dominates antimatter?  Dark matter: where is concentrated the hidden mass of Universe?  World geometry: are there extra dimensions? bulk volume geometry?  New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)  QCD matter: what are the properties of QCD at extreme high densities, does QGP exist? Standard model: validity bounds -> high precision tests of SM ! Open Questions:  Mass: why do fundamental particle have mass and why these masses are such different?  Antimatter: why at the present days matter dominates antimatter?  Dark matter: where is concentrated the hidden mass of Universe?  World geometry: are there extra dimensions? bulk volume geometry?  New particles: do they exist? (sparticles, extra gauge bosons, graviton states etc)  QCD matter: what are the properties of QCD at extreme high densities, does QGP exist? Standard model: validity bounds -> high precision tests of SM ! LHC Physics Perspectives 5 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

6. Compact Muon Solenoid 6 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

7. 7 Compact Muon Solenoid (CMS) + TOTEM (TOTal Elastic and diffractive cross section Measurement) A Toroidal LHC ApparatuS (ATLAS) A Large Ion Collider Experiment (ALICE) LHC-B LHC Experiments LHCf

8. CMS at LHC CMS CERN Building 40 14 km 15 min 8 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

9. Compact Muon Solenoid - CMS Muon ChambersTrackerElectromagnetic calorimeter, ECAL Hadron calorimeter HCAL Forward Calorimeter, HF Superconducting Coil diameter 6 m, length 13 m Return Yoke Silicon Tracker PbWO 4 ECAL sampling brass HCAL MUON Chambers Detector subsystems are designed to measure: the energy and momentum of photons, electrons, muons, jets, missing E T up to a few TeV Weight 12 500 t Diameter 15.00 m Length 21.6 m Magnetic Field 3.9 Tesla 9 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

10. Russian and Dubna Member State CMS Collaboration CERN France Italy UK Switzerland USA Austria Finland Greece Hungary Belgium Poland Portugal Spain Pakistan Georgia Armenia Ukraine Uzbekistan Cyprus Croatia China, PR Turkey Belarus Estonia India Germany Korea Russia and JINR Member States Bulgaria China (Taiwan) Iran Serbia New-Zealand Brazil Ireland Mexico Colombia Lithuania RDMS CMS: 2310 authors 38 countries 175 institutions the RDMS CMS Collaboration was founded in Dubna in September 1994 10 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

11. JINR Participation in CMS Construction JINR participates in the CMS in a framework of the RDMS CMS Collaboration RDMS bears Full Responsibility JINR Participates SE HE ME1/1 11 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

12. Cross-sections of physics processes at LHC Energy The fraction of “important” events is ~ 10 -8 of total event flow ! The fraction of events from “new physics” is 10 -10 - 10 -15 (Higgs, SUSY, BSM) A very small part of events is a proper for physics analysis No opportunity to store and to process the full data flow Triggering and Selection 12 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

13. Online: Trigger System Multilevel trigger rejection of background (filtration of events) reduction of data volume on-line reduction factor ~10 7 Trigger menu selection of required events The data flow is reduced from 1000 TB/sec (1TB=10 12 Bytes) at input of the first-level trigger down to 100- 200 MB/sec on output of the third- level trigger (to record on disks or tapes for further analysis) Level 1 - Special Hardware Level 2 - Embedded Processors 40 MHz (1000 TB/sec) Level 3 – Farm of commodity CPUs 75 KHz (75 GB/sec) 5 KHz (5 GB/sec) 100 Hz (100 MB/sec) Data Recording & Offline Analysis CMS event – 1-2 MB L1 trigger – 75 GB/с HLT trigger – 0.1 GB/с total flow ~ 3 PByear 13 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

14. LHC Data Flows ~ 3 PB/year 14 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

15. The CMS Computing Requests Event rate to tape: 100 Hz (10 9 events/year) Days of running/year: 200 Data written to tape: ~ 3 PetaBytes/Year Monte Carlo events: 9∙10 8 CPU (Tier-0 + Tier-1s + Tier-3s): ~ 26 000 kSI2k Typical networks: 1 - 10 Gbps Users: ~ 1000 physicists distributed all over the world 1 billion people surfing the Web 10 5 10 4 10 3 10 2 Level 1 Rate (Hz) High Level-1 Trigger (1 MHz) High No. Channels High Bandwidth (500 Gbit/s) High Data Archive (PetaByte) LHCB KLOE HERA-B CDF II CDF H1 ZEUS UA1 LEP NA49 ALICE Event Size (bytes) 10 4 10 5 10 6 ATLAS CMS 10 6 10 7 The full processing and analysis can not be performed by the standard tool kits It is essential to use Distributed Computer Resources 15 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

16. CMS Computing Model 16 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

17. 17 Tier-0 (CERN): accepts data from the CMS Online Data Acquisition and Trigger System (TriDAS) archives RAW data the first pass of reconstruction and performs Prompt Calibration data distribution to Tier-1 Tier-1 (11 centers): receives a data from the Tier-0 data processing (re-reconstruction, skimming, calibration etc) distributes data and MC to the other Tier-1 and Tier-2 secure storage and redistribution for data and MC Tier-2 (>200 centers): simulation user physics analysis Tier Structure of GRID Distributed Computing: Tier-0/Tier-1/Tier-2/Tier-3 17 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

18. CMS GRID Distributed Computing Tier-0 at CERN Tier-1 (7 centers): CERN, KIT (Germany), PIC (Spain), CCIN2P3 (France), CNAF (Italy), ASGC (Taipei), RAL (UK), FNAL (US) + two sites under construction at Dubna and India Tier-2 (85 centers): 7 in Russian and DMS 18 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

19. CMS Data Model Event FormatContentsPurposeEvent Size (MB) DAQ-RAWDetector data from front end electronics + L1 trigger result.Primary record of physics event. Input to online HLT 1-1.5 RAWDetector data after online formatting, the L1 trigger result, the result of the HLT selections (HLT trigger bits Input to Tier-0 reconstruction. Primary archive of events at CERN. 1.5 RECOReconstructed objects (tracks, vertices, jets, electrons, muons, etc.) and reconstructed hits/clusters Output of Tier-0 reconstruction and subsequent rereconstruction passes.. 0.25 AODSubset of RECO. Reconstructed objects (tracks, vertices, jets, electrons, muons, etc Physics analysis, limited refitting of tracks and clusters 0.05 TAGRun/event number, high-level physics objects, used to indexRapid identification of events0.01 FEVTFull Event: Term used to refer to RAW+RECO togethermultiple1.75 GENGenerated Monte Carlo event-- SIMEnergy depositions of MC particles in detector (sim hits).-- DIGISim hits converted into detector response (the same as the RAW)-1.5 19 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

20. ~ Pbyte/sec 10 Gbps ~ 100-1500 MBytes/sec 10 Gbps 0.1 - 10 Gbps 2-10 Gbps CMS Data Flows (I) 20 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

21. CMS Data Flows (II) T0 to T1: scheduled, time-critical, continuous during data-taking reliable transfer needed for fast a access to new data, and to ensure that data is stored safely T1 to T1: redistributing data, generally after reprocessing (e.g. processing with improved algorithms) T1 to T2: Data for analysis at Tier-2s Detector data flow Monte-Carlo data flow T2 to T1: data transfer simulation events T1 to T1/T2: data transfer re-processed simulation events (e.g. processing with improved algorithms) 21 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

22. Data Processing (II) Calibration Data Base RAW Data Reconstruction Event Selection Physics Analysis Theory/ Monte Carlo Simulation CMS-PAS-HIG-11-032 Final Physics Results! 22 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

23. CMS Data Processing (II) 23 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

24. CMS Computing Resources 24 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

25. RDMS CMS Computing Model 25 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

26. RDMS CMS Grid Sector CERN Tier-0 CMS RDMS Tier-1 RDMS Grid Sector Russia DMS Tier-2 JINR Dubna Tier-2 ITEP Moscow Tier-2 SINP MSU Moscow Tier-2 INR Troitsk Tier-2 FIAN Moscow Tier-2 IHEP Protivino Tier-2 PNPI St.Petersburg Tier-2 KIPT Kharkov Tier-2 Minsk Tier-2 Erevan Tier-2 Sofia Tier-2 Tbilisi To provide data management and data transfer to RDMS Tier-2 and data processing in the framework of RDMS Physics Tasks the special СMS RDMS Tier-1 was created in CERN Dubna CMS Tier-1 and other CMS Tier-1 centres 26 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

27. CMS T2 requirements Basic requirements to CMS VO T2 sites for Physics group hosting: info on contact persons responsible for site operation site visibility (BDII) and CMS VO support availability of CMSSW actual version daily SAM availability > 80% daily CMS job robot efficiency > 90% regular file transfer test “OK” Certified links with CMS T1:  2 up and  4 down disk space ~ 150-200 TB for: - central space (~30 TB) - analysis space (~60-100 TB) - MC space (~20 TB) - local space (~30-60 TB) - local CMS users space (~2 TB per user) CPU resources ~ 3-4 KSI2K per 1 TB disk space, 2GB RAM memory per job 27 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

28. RDMS CMS Tier-2 Availability 28 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

29. Status of RDMS Site Metrics 29 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

30. Normalized CPU time (in 1K.SI2K.Hours) consumed by the CMS sites September, 2010 – August, 2011) In total 283,729,684 K.SI2K.Hours 30 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

31. Total Number of CMS Jobs at RDMS CMS sites 1 747 182 jobs 2 253 116 jobs 5.8% 8.7% 31 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

32. Job Summary at RDMS CMS Sites as in the CMS Dashboard Job Status per Sites Site ActivityJob Efficiency 32 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

33. PHEDEX Transfer Rate & Transfer Volumes for RDMS CMS sites 33 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

34. RDMS CMS T2 association Now Future interest Analysis Groups Exotica: T2_RU_JINR Exotica: T2_RU_INR HI: T2_RU_SINP QCD: T2_RU_PNPI Electroweak: T2_UA_KIPT Top: T2_RU_SINP FWD: T2_RU_IHEP Object/Performance Groups Muon: T2_RU_JINR e-gamma-ECAL: T2_RU_INR JetMET-HCAL: T2_RU_ITEP 34 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

35. JINR Monitoring and Analysis Remote Centre  Monitoring of detector systems  Data Monitoring / Express Analysis  Shift Operations (except for run control)  Communications of JINR shifter with personal at CMS Control Room (SX5) and CMS Meyrin centre  Communications between JINR experts and CMS shifters  Coordination of data processing and data management  Training and Information 35 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

36. RDMS CMS Computing Model: Tier-1 at Dubna 36 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

37. WLCG Tier1 center in Russia Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011). The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011) - for all four experiments ALICE, ATLAS, CMS and LHCb - ~10% of the summary Tier1 (without CERN) resources - increase by 30% each year - draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of next working LHC session. Proposal to create the LCG Tier1 center in Russia (official letter by Minister of Science and Education of Russia A. Fursenko has been sent to CERN DG R. Heuer in March 2011). The corresponding point to include in the agenda of next 5x5 meeting Russia-CERN (October 2011) - for all four experiments ALICE, ATLAS, CMS and LHCb - ~10% of the summary Tier1 (without CERN) resources - increase by 30% each year - draft planning (proposal under discussion) to have prototype in the endof 2012, and full resources in 2014 to meet the start of next working LHC session. 37 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

38. Joint NRC "Kurchatov Institute" – JINR Tier1 Computing Centre Project: «Creation of the automated system of data processing for experiments at the Large Hadron Collider (LHC) of Tier1 level and maintenance of Grid-services for a distributed analysis of these data» Terms: 2011-2013 Type of project: R&D Cost: federal budget - 280 million rubles (~8.5 MCHF), extrabudgetary sources - 50% of the total cost Leading executor: RRC KI «Kurchatov institute» for ALICE, ATLAS, and LHC-B Co-executor: LIT JINR (Dubna) for the CMS experiment Project goal: creation in Russia of a computer-based system for processing experimental data received at the LHC and provision of Grid-services for a subsequent analysis of these data at the distributed centers of the LHC global Grid- system. Core of the proposal: development and creation of a working prototype of the first-level center for data processing within the LHC experiments with a resource volume not less than 15% of the required one and a full set of grid-services for a subsequent distributed analysis of these data. 38 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

40. 40 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

41. Tier-1 Services Security (GSI) Computing Element (CE) Storage Element (SE) Monitoring and Accounting Virtual Organizations (VOMS) Workload management (WMS) Information service (BDII) File transfer service (FTS + PhEDEx) VOBOX SQUID / Frontier Servers CMS user services (Reconstruction Services, Analysis Services etc) 41 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

42. CMS Tier-1 Requirements input flow ~ 7 GB/s output flow ~ 3-4 GB/s CPU – 2.5 MSI2k disks – 1.2 PB tapes for archives and secure store – 2.8 PB data loss – 10 GB per 1 PB access to data ~ 800 MB/s I/O computing node bandwidth ~ GB 42 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

43. JINR Resources JINR-LCG2 Tier2 Sept. 2011April 2012Dec. 2012Dec. 2013 CPU (HEP-SPEC06) 16 00020 00030 000 45 000 Disk(Tbytes) CMS ATLAS ALICE 635 240 275 995 360 275 1800 660 480 3000 1100 800 Tape (Tbytes)---5000 JINR-CMS Tier1 Dec. 2012 Prototype Sep. 2014 Start CPU (HEP-SPEC06) 4 00045 000 Disk(Tbytes) CMS 5008 000 Tape (Tbytes)-12 000 43 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

44. SUMMARY RDMS Tier-2 sites are fully satisfied to CMS Computing Requirements  are involved actively into data processing and analysis (~3.8% of all CMS resources) New CMS Tier-1 site is under construction in Dubna (in 2012-2013 a prototype need to be designed and commmisined) CMS Regional Operation Center is founded and operated in JINR for remote monitoring of detector systems and data express- analysis