1. GRID in JINR and participation in the WLCG project Korenkov Vladimir LIT, JINR 17.07.12

2. T.Strizh (LIT, JINR) Tier Structure of GRID Distributed Computing: Tier-0/Tier-1/Tier-2/Tier-3 Ian.Bird@cern.ch 2 Tier-0 (CERN): accepts data from the CMS Online Data Acquisition and Trigger System archives RAW data the first pass of reconstruction and performs Prompt Calibration data distribution to Tier-1 Tier-1 (11 centers): receives a data from the Tier-0 data processing (re- reconstruction, skimming, calibration etc) distributes data and MC to the other Tier-1 and Tier-2 secure storage and redistribution for data and MC Tier-2 (>200 centers): simulation user physics analysis

3. T.Strizh (LIT, JINR) Some history 1999 – Monarc Project 1999 – Monarc Project Early discussions on how to organise distributed computing for LHCEarly discussions on how to organise distributed computing for LHC 2001-2003 - EU DataGrid project 2001-2003 - EU DataGrid project middleware & testbed for an operational gridmiddleware & testbed for an operational grid 2002-2005 – LHC Computing Grid – LCG 2002-2005 – LHC Computing Grid – LCG deploying the results of DataGrid to provide adeploying the results of DataGrid to provide a production facility for LHC experiments 2004-2006 – EU EGEE project phase 1 2004-2006 – EU EGEE project phase 1 starts from the LCG gridstarts from the LCG grid shared production infrastructureshared production infrastructure expanding to other communities and sciencesexpanding to other communities and sciences 2006-2008 – EU EGEE-II 2006-2008 – EU EGEE-II Building on phase 1Building on phase 1 Expanding applications and communities …Expanding applications and communities … 2008-2010 – EU EGEE-III 2008-2010 – EU EGEE-III 2010-2012 - EGI-InSPIRE 2010-2012 - EGI-InSPIRE CERN

4. T.Strizh (LIT, JINR) Russian Data Intensive Grid infrastructure (RDIG) RDIG Resource Centres: – ITEP – JINR-LCG2 (Dubna) – RRC-KI – RU-Moscow-KIAM – RU-Phys-SPbSU – RU-Protvino-IHEP – RU-SPbSU – Ru-Troitsk-INR – ru-IMPB-LCG2 – ru-Moscow-FIAN – ru-Moscow-MEPHI – ru-PNPI-LCG2 (Gatchina) – ru-Moscow-SINP - Kharkov-KIPT (UA) - BY-NCPHEP (Minsk) - UA-KNU -- UA-BITP The Russian consortium RDIG (Russian Data Intensive Grid), was set up in September 2003 as a national federation in the EGEE project. Now the RDIG infrastructure comprises 17 Resource Centers with > 20000 kSI2K CPU and > 4500 TB of disc storage.

5. T.Strizh (LIT, JINR) CPU: 2560 Core Total performance 24000 HEPSpec06 Disk storage capacity 1800 TB Growth of the JINR Tier2 resources in 2003 - 2011 JINR Tier2 Center Availability and Reliability = 99% HEPSpec06 ) More than 7,5 millions tasks and more than 130 millions Normalised CPU time ( HEPSpec06 ) were executed during July 2011- July 2012 Performance Disk storage capacity

6. T.Strizh (LIT, JINR) Scheme of the CICC network connections

7. T.Strizh (LIT, JINR) Support of VOs and experiments Now CICC JINR as a grid-site of the global grid infrastructure supports computations of 10 Virtual Organizations (alice, atlas, biomed, cms, dteam, fusion, hone, lhcb, rgstest and ops), and also gives a possibility of using the grid- resources for the CBM and PANDA experiments.

8. T.Strizh (LIT, JINR) JINR monitoring main page JINR LAN Infrastructure Monitoring is one of the most important services. The Network monitoring information system (NMIS) with web-based graphical interface – http://litmon.jinr.ru provides the operational monitoring processes and monitors the mosthttp://litmon.jinr.ru important network elements: JINR LAN routers and switches, remote routers and switches, cooling units, DNS servers, dCache servers, Oracle servers, RAIDs, WLCG servers, working nodes, etc. More than 350 network nodes are in round-the-clock monitoring.

9. T.Strizh (LIT, JINR) 9 Country Normalized CPU time per Country LHC VO (July 2011 - July 2012) All Country - 3,808,660,150 Russia- 95,289,002 (2.5%) 2011 - 2,108,813,533 Russia - 45,385,987

10. T.Strizh (LIT, JINR) 1010 Russia Normalized CPU time per SITE LHC VO (July 2011 - July 2012)

11. Contribution of RDMS CMS sites – 4% (16,668,792 kSI2K ) Contribution of JINR – 52% CMS Computing Support at the JINR CMS T2 center at JINR: CPU1240kSI2k Slots495jobs Storage Disk376TB All CMS Sites CMS Sites in Russia JINR Statistics for 12 last months: 08/2011-07/2012

12. T.Strizh (LIT, JINR) JINR Monitoring and Analysis Remote Centre  Monitoring of detector systems  Data Monitoring / Express Analysis  Shift Operations (except for run control)  Communications of JINR shifter with personal at CMS Control Room (SX5) and CMS Meyrin centre  Communications between JINR experts and CMS shifters  Coordination of data processing and data management  Training and Information 12 Ilya Gorbunov and Sergei Shmatov “RDMS CMS Computing”, AIS2012, Dubna, May 18, 2012

13. T.Strizh (LIT, JINR) Atlas Computing Support at JINR Atlas at JINR (Statistics for 12 last months: 07.2011- 07.2012) : CPU time (HEPSpec06): 51,948,968 (52%) Jobs : 4,885,878 (47%) Atlas Sites in Russia

14. T.Strizh (LIT, JINR) System of remote access in real time (SRART) for monitoring and quality assessment of data from the ATLAS at JINR At present the system of remote access in real time is debugged on real data of the ATLAS experiment. One of the most significant results of the team TDAQ ATLAS at LIT during the last few years was the participation in the development of the project TDAQ ATLAS at CERN. The system of remote access in real time (SRART) for monitoring and quality assessment of data from the ATLAS at JINR was put in operation.

15. T.Strizh (LIT, JINR) ALICE Computing Support at JINR Seven RDIG sites (IHEP, ITEP, JINR, PNPI, RRC-KI, SPSU and Troitsk) in last 6 months contribute 4.19% to whole CPU of ALICE; -”- 9.18% to whole DONE Jobs number in last 6 months Delivered Efficiency Job statistics Group CPU Wall CPU/Wall Assigned Completed Efficiency RDIG 1293 2395 54.02% 717047 528445 73.7% In July 2012

16. T.Strizh (LIT, JINR) SE usage More than 1PB of disk space are allocated today by seven RDIG sites for ALICE. More than 1PB of disk space are allocated today by seven RDIG sites for ALICE. ~744 TB of this disk space is used today. ALICE management asks about increase disk capacity as a speed of new data taken and development is much faster than speed of disk cleaning. ~744 TB of this disk space is used today. ALICE management asks about increase disk capacity as a speed of new data taken and development is much faster than speed of disk cleaning. Example of CPU usage share between seven RDIG sites. Example of CPU usage share between seven RDIG sites.

17. T.Strizh (LIT, JINR) Aggregate SE traffic Summary SE in traffic in 6 months was = 943 TB, out was = 9.6PB

18. T.Strizh (LIT, JINR) The tasks of the Russia & JINR in the WLCG (2012 years): Task 1. MW (gLite) testing (supervisor O. Keeble) Task 1. MW (gLite) testing (supervisor O. Keeble) Task 2. LCG vs Experiments (supervisor I. Bird) Task 2. LCG vs Experiments (supervisor I. Bird) Task 3. LCG monitoring (supervisor J. Andreeva) Task 3. LCG monitoring (supervisor J. Andreeva) Task 4. Tier3 monitoring (supervisor J. Andreeva, A.Klementov) Task 4. Tier3 monitoring (supervisor J. Andreeva, A.Klementov) Task 5/6. Genser/ MCDB ( supervisor W. Pokorski) Task 5/6. Genser/ MCDB ( supervisor W. Pokorski) Task 7. Tier1 (supervisor I. Bird) Task 7. Tier1 (supervisor I. Bird) Worldwide LHC Computing Grid Project (WLCG) The protocol between CERN, Russia and JINR on participation in LCG Project was approved in 2003. MoU on Worldwide LHC Computing Grid (WLCG) signed by Russia and JINR in October, 2007

19. T.Strizh (LIT, JINR) 19 The tasks of the JINR in the WLCG: WLCG-infrastructure support and development at JINR; WLCG-infrastructure support and development at JINR; participation in WLCG middleware testing/evaluation, participation in WLCG middleware testing/evaluation, grid monitoring and accounting tools development; grid monitoring and accounting tools development; Development software for HEP applications; Development software for HEP applications; MCDB development; MCDB development; support of Users, Virtual Organization (VO) and application; User & Administrator training and education; User & Administrator training and education; support of JINR Member States in the WLCG activities; support of JINR Member States in the WLCG activities; Worldwide LHC Computing Grid Project (WLCG)

20. T.Strizh (LIT, JINR) 20 RDIG monitoring&accounting http://rocmon.jinr.ru:8080 Monitored values CPUs - total /working / down/ free / busy Jobs - running / waiting Storage space - used / available Network - Available bandwidth Accounting values Number of submitted jobs Used CPU time Totally sum in seconds Normalized (with WNs productivity) Average time per job Waiting time Totally sum in seconds Average ratio waiting/used CPU time per job Physical memory Average per job JINR CICC Monitoring – allows to keep an eye on parameters of Grid sites' operation in real time Accounting - resources utilization on Grid sites by virtual organizations and single users

21. T.Strizh (LIT, JINR) Architecture

22. 22 monitoring system FTS current stateanalysis of errorsfunctionality and optimization reliability and performance A monitoring system is developed which provides a convenient and reliable tool for receiving detailed information about the FTS current state and the analysis of errors on data transfer channels, maintaining FTS functionality and optimization of the technical support process. The system could seriously improve the FTS reliability and performance. FTS (FILE TRANSFER SERVICE) MONITORING FOR WORLDWIDE LHC COMPUTING GRID (WLCG) PROJECT

23. T.Strizh (LIT, JINR) The Worldwide LHC Computing Grid (WLCG)

24. T.Strizh (LIT, JINR) ATLAS DQ2 Deletion service During the 2010 year the works on development of the deletion service for ATLAS Distributed Data Management (DDM) system were performed During the 2010 year the works on development of the deletion service for ATLAS Distributed Data Management (DDM) system were performed Works started at the middle of April 2010. At the end August new version of Deletion Service was tested for set of sites and from November of 2010 for all sites managed by DQ2 Works started at the middle of April 2010. At the end August new version of Deletion Service was tested for set of sites and from November of 2010 for all sites managed by DQ2 Development comprises the building of new interfaces between parts of deletion service (based on the web service technology), creating new database schema, rebuilding the deletion service core part, development of extended interfaces with mass storage systems and extension of the deletion monitoring system. Development comprises the building of new interfaces between parts of deletion service (based on the web service technology), creating new database schema, rebuilding the deletion service core part, development of extended interfaces with mass storage systems and extension of the deletion monitoring system. Deletion service maintained by JINR specialists. Deletion service maintained by JINR specialists.

25. T.Strizh (LIT, JINR) Tier 3 sites monitoring project Traditional LHC Distributed Computing Tier-0 (CERN) → Tier-1 → Tier-2 Additional → Tier-3 Needs → the global view of the LHC computing activities The LIT participates in the development of a software suite for Tier-3 sites monitoring A virtual testbed has been created at JINR which allows simulation of various Tier3 clusters and solutions for data storage. Tier-0 Tier-1 Tier-2 Tier-3 CERN Analysis Facility Raw Raw/AOD/ESD AOD DPD Interactive analysis plots, fits, toy MC, studies, … 12 Sites Worldwide 120+ Sites Worldwide development host 2 WNs Headnode + WN PBS (Torque) 2 servers Manager (redirector) XRootD 2 WNs Headnode PROOF 2 WNs Headnode + WN Condor 2 WNs Headnode Oracle Grid Engine 2 servers Manager (redirector) XRootD OSS Client Lustre MDS Ganglia web frontend

26. T.Strizh (LIT, JINR) Tier 3 sites monitoring project (2011) Tier-3 sites consist of resources mostly dedicated for the data analysis by the geographically close or local scientific groups. Set of Tier 3 sites can be joined to federation. Tier-3 sites consist of resources mostly dedicated for the data analysis by the geographically close or local scientific groups. Set of Tier 3 sites can be joined to federation. Many Institutes and National Communities built (or have plans to build) Tier-3 facilities. Tier-3 sites comprise a range of architectures and many do not possess Grid middleware, which would render application of Grid monitoring systems useless. Many Institutes and National Communities built (or have plans to build) Tier-3 facilities. Tier-3 sites comprise a range of architectures and many do not possess Grid middleware, which would render application of Grid monitoring systems useless. Joined effort of ATLAS, BNL (USA), JINR and CERN IT (ES group) Joined effort of ATLAS, BNL (USA), JINR and CERN IT (ES group) Objectives for Tier3 monitoring Objectives for Tier3 monitoring Monitoring of Tier 3 site.Monitoring of Tier 3 site. Monitoring of Tier 3 sites federation.Monitoring of Tier 3 sites federation. Monitoring of Tier 3 site Monitoring of Tier 3 site Detailed monitoring of the local fabric (overall cluster or clusters monitoring, monitoring each individual node in the cluster, network utilization)Detailed monitoring of the local fabric (overall cluster or clusters monitoring, monitoring each individual node in the cluster, network utilization) Monitoring of the batch system.Monitoring of the batch system. Monitoring of the mass storage system (total and available space, number of connections, I/O performance)Monitoring of the mass storage system (total and available space, number of connections, I/O performance) Monitoring of VO computing activities at a siteMonitoring of VO computing activities at a site Monitoring of Tier 3 sites federation Monitoring of Tier 3 sites federation Monitoring of the VO usage of the Tier3 resources in terms of data transfer and job processing and the quality of the provided service based on the job processing and data transfer monitoring metrics.Monitoring of the VO usage of the Tier3 resources in terms of data transfer and job processing and the quality of the provided service based on the job processing and data transfer monitoring metrics.

27. T.Strizh (LIT, JINR) xRootD monitoring architecture Implementation of the XROOTD monitoring foresees 3 levels of hierarchy: site, federation, global. Implementation of the XROOTD monitoring foresees 3 levels of hierarchy: site, federation, global. Monitoring on the site level is implemented in the framework of the Tier3 monitoring project and is currently under validation by the ATLAS pilot sites. Monitoring on the site level is implemented in the framework of the Tier3 monitoring project and is currently under validation by the ATLAS pilot sites. The site collector reads xrootd summary and detailed flows, process them in order to reformat into event-like information and publishes reformatted info to MSG. At the federation level data consumed from MSG is processed using hadoop/mapreduce to correlate events which are related to the same transfer and to generate federation level statistics which becomes available via UI and APIs. The UI is similar to the UI of the Global WLCG transfer Dashboard, therefore most of the UI code can be reused. Finally messages similar to FTS transfer messages describing every particular transfer are published from the federation to MSG to be further consumed by Global WLCG transfer Dashboard. The site collector reads xrootd summary and detailed flows, process them in order to reformat into event-like information and publishes reformatted info to MSG. At the federation level data consumed from MSG is processed using hadoop/mapreduce to correlate events which are related to the same transfer and to generate federation level statistics which becomes available via UI and APIs. The UI is similar to the UI of the Global WLCG transfer Dashboard, therefore most of the UI code can be reused. Finally messages similar to FTS transfer messages describing every particular transfer are published from the federation to MSG to be further consumed by Global WLCG transfer Dashboard. 27

28. T.Strizh (LIT, JINR) xRootD monitoring: the schema of information flow 28

29. T.Strizh (LIT, JINR) Data transfer global monitoring system 29

30. T.Strizh (LIT, JINR) 30 Training courses on grid technologies Training courses on grid technologies for users (basic idea and skills), for users (basic idea and skills), for system administrators (grid sites deployment), for system administrators (grid sites deployment), for grid applications and services developers. for grid applications and services developers. Evaluation of different implementations of grid middleware and testing particular grid services. Evaluation of different implementations of grid middleware and testing particular grid services. Grid services development and porting existing applications into grid environment. Grid services development and porting existing applications into grid environment. Grid training and education (T-infrastructure)

31. T.Strizh (LIT, JINR) 31 T-infrastructure implementation All services are deployed on VMs (OpenVZ) All services are deployed on VMs (OpenVZ) Main parts: Main parts: three grid sites on gLite middleware, three grid sites on gLite middleware, GT5 testbed, GT5 testbed, desktop grid testbed based on BOINC, desktop grid testbed based on BOINC, testbed for WLCG activities. testbed for WLCG activities. Running since 2006 Running since 2006

32. T.Strizh (LIT, JINR) JINR Grid-infrastructure for training and education – first step towards construction of the JINR Member States grid-infrastructure Consists of three grid sites at JINR and one at each of the following sites: Institute of High-Energy Physics - IHEP (Protvino), Institute of High-Energy Physics - IHEP (Protvino), Institute of Mathematics and Information Technologies AS of Republic of Uzbekistan - IMIT (Tashkhent, Uzbekistan), Institute of Mathematics and Information Technologies AS of Republic of Uzbekistan - IMIT (Tashkhent, Uzbekistan), Sofia University "St. Kliment Ohridski" - SU (Sofia, Bulgaria), Sofia University "St. Kliment Ohridski" - SU (Sofia, Bulgaria), Bogolyubov Institute for Theoretical Physics - BITP (Kiev, Ukraine), Bogolyubov Institute for Theoretical Physics - BITP (Kiev, Ukraine), National Technical University of Ukraine "Kyiv Polytechnic Institute" - KPI (Kiev, Ukraine). National Technical University of Ukraine "Kyiv Polytechnic Institute" - KPI (Kiev, Ukraine). Letters of Intent with Moldova “MD-GRID”, Mongolia “Mongol- Grid”, Kazakhstan Project with Cairo University https://gridedu.jinr.ru

33. T.Strizh (LIT, JINR) 33 Participation in GridNNN project Grid support for Russian national nanotechnology network Grid support for Russian national nanotechnology network To provide for science and industry an effective access to the distributed computational, informational and networking facilities To provide for science and industry an effective access to the distributed computational, informational and networking facilities Expecting breakthrough in nanotechnologies Expecting breakthrough in nanotechnologies Supported by the special federal program Supported by the special federal program Main points Main points based on a network of supercomputers (about 15-30) based on a network of supercomputers (about 15-30) has two grid operations centers (main and backup) has two grid operations centers (main and backup) is a set of grid services with unified interface is a set of grid services with unified interface partially based on Globus Toolkit 4 partially based on Globus Toolkit 4

34. T.Strizh (LIT, JINR) 34 GridNNN infrastructure 10 resource centers at the moment in different regions of Russia RRC KI, «Chebyshev» (MSU), IPCP RAS, CC FEB RAS, ICMM RAS, JINR, SINP MSU, PNPI, KNC RAS, SPbSU RRC KI, «Chebyshev» (MSU), IPCP RAS, CC FEB RAS, ICMM RAS, JINR, SINP MSU, PNPI, KNC RAS, SPbSU

35. T.Strizh (LIT, JINR) Russian Grid Network Goal — to make a computational base for hi-tech industry and science Goal — to make a computational base for hi-tech industry and science Using the network of supercomputers and original software created within recently finished GridNNN project Using the network of supercomputers and original software created within recently finished GridNNN project Some statistics Some statistics 19 resource centers 19 resource centers 10 virtual organizations 10 virtual organizations 70 users 70 users more than 500'000 more than 500'000 tasks processed

36. T.Strizh (LIT, JINR) Project: Model of a shared distributed system for acquisition, transfer and processing of very large- scale data volumes, based on Grid technologies, for the NICA accelerator complex Terms: 2011-2012 Cost: federal budget - 10 million rubles, extrabudgetary sources - 25% of the total cost Leading executor: LIT JINR Co-executor: VBLHEP JINR MPD data processing model ( from “The MultiPurpose Detector – MPD Conceptual Design Report v. 1.4 ”)

37. T.Strizh (LIT, JINR) 37 WLCG Tier1 in Russia Proposal to create the LCG Tier-1 center in Russia: official letter by A. Fursenko, Minister of Science and Education of Russia, has been sent to CERN DG R. Heuer in March 2011 Proposal to create the LCG Tier-1 center in Russia: official letter by A. Fursenko, Minister of Science and Education of Russia, has been sent to CERN DG R. Heuer in March 2011 The corresponding points were included to the agenda of Russia – CERN 5x5 meeting in 2011 The corresponding points were included to the agenda of Russia – CERN 5x5 meeting in 2011 serving all four experiments: ALICE, ATLAS, CMS and LHCb serving all four experiments: ALICE, ATLAS, CMS and LHCb ~10% of the total existing Tier1 resources (without CERN) ~10% of the total existing Tier1 resources (without CERN) increase by 30% on each year increase by 30% on each year draft planning (proposal under discussion) to have prototype in the end of 2012 and full resources in 2014 to be ready for the start of the next LHC session draft planning (proposal under discussion) to have prototype in the end of 2012 and full resources in 2014 to be ready for the start of the next LHC session

38. T.Strizh (LIT, JINR) JINR Resources JINR-LCG2 Tier2 April 2012Dec. 2012Dec. 2013 CPU (HEP-SPEC06) 20 00030 000 45 000 Disk(Tbytes) CMS ATLAS ALICE 995 360 275 1800 660 480 3000 1100 800 Tape (Tbytes)--5000 JINR-CMS Tier1 Dec. 2012 Prototype Sep. 2014 Start CPU (HEP-SPEC06) 5 00050 000 Disk(Tbytes) CMS 5008 000 Tape (Tbytes)50015 000

39. T.Strizh (LIT, JINR) 39 Frames for Grid cooperation of JINR Worldwide LHC Computing Grid (WLCG); EGI-InSPIRE EGI-InSPIRE RDIG Development Global data transfer monitoring system for WLCG infrastructure CERN-RFBR project “Global data transfer monitoring system for WLCG infrastructure” NASU-RFBR project “Development and support of LIT JINR and NSC KIPT grid- infrastructures for distributed CMS data processing of the LHC operation” NASU-RFBR project “Development and support of LIT JINR and NSC KIPT grid- infrastructures for distributed CMS data processing of the LHC operation” BMBF grant “Development of the Grid-infrastructure and tools to provide joint investigations performed with participation of JINR and German research centers” “Development of Grid segment for the LHC experiments” was supported in frames of JINR-South Africa cooperation agreement; Development of Grid segment at Cairo University and its integration to the JINR GridEdu infrastructure Development of Grid segment at Cairo University and its integration to the JINR GridEdu infrastructure JINR - FZU AS Czech Republic Project “The GRID for the physics experiments” JINR - FZU AS Czech Republic Project “The GRID for the physics experiments” JINR-Romania cooperation Hulubei-Meshcheryakov programme (MD-GRID, RENAM) JINR-Moldova cooperation (MD-GRID, RENAM) (Mongol-Grid) JINR-Mongolia cooperation (Mongol-Grid) JINR-Slovakia cooperation Kazakhstan JINR- Kazakhstan cooperation (ENU Gumelev) Russian Grid Network” Project “Russian Grid Network”

40. T.Strizh (LIT, JINR) WEB-PORTAL “GRID AT JINR” – “ГРИД В ОИЯИ”: http://grid.jinr.ru A new informational resource has been created at JINR: web-portal “GRID AT JINR”. The content includes the detailed information on the JINR grid-site and JINR’s participation in grid projects: КОНЦЕПЦИЯ ГРИД ГРИД-технологии ГРИД-проекты Консорциум РДИГ ГРИД-САЙТ ОИЯИ Инфраструктура и сервисы Схема Статистика Поддержка ВО и экспериментов ATLAS CMS CBM и PANDA HONE Как стать пользователем ОИЯИ В ГРИД-ПРОЕКТАХ WLCG ГридННС EGEE Проекты РФФИ Проекты ИНТАС СКИФ-ГРИД ТЕСТИРОВАНИЕ ГРИД-ПО СТРАНЫ-УЧАСТНИЦЫ ОИЯИ МОНИТОРИНГ И АККАУНТИНГ RDIG-мониторинг dCache-мониторинг Dashboard FTS-мониторинг Н1 МС-мониторинг ГРИД-КОНФЕРЕНЦИИ GRID NEC ОБУЧЕНИЕ Учебная грид-инфраструктура Курсы и лекции Учебные материалы ДОКУМЕНТАЦИЯ Статьи Учебные материалы НОВОСТИ КОНТАКТЫ