1. А.Минаенко Совещание по физике и компьютингу, 03 февраля 2010 г. НИИЯФ МГУ, Москва Текущее состояние и ближайшие перспективы компьютинга для АТЛАСа в России

2. Layout Ru-Tier-2 tasks Computing resources in 2009 and their increase for 2010 NL Tier-1 CPU resources usage in 2009 RuTier-2 and data distribution STEP09 lessons 03/02/2010A.Minaenko2

3. ATLAS RuTier-2 tasks Russian Tier-2 (RuTier-2) computing facility is planned to supply with computing resources all 4 LHC experiments including ATLAS. It is a distributed computing center including computing farms of 6 institutions: ITEP, RRC-KI, SINP (all Moscow), IHEP (Protvino), JINR (Dubna), PNPI (St.Petersburg) Recently two smaller sites MEPhI and FIAN have been added The main RuTier-2 task is providing facilities for physics analysis of collected data using mainly AOD, DPD and user derived data formats It includes also development of reconstruction algorithms using limited subsets of ESD and Raw data 50 active ATLAS users are supposed to carry on physics data analysis at RuTier-2 Now group ru is created in the framework of ATLAS VO. It includes physicists intending to carry analysis in RuTier-2 and the group list contains 38 names at the moment. The group will have privilege of write access to local RuTier-2 disk resources (space token LOCALGROUPDISK) All the data used for analysis should be stored on disks The second important task is production and storage of MC simulated data The full size of data and CPU needed for their analysis are proportional to the collected statistics. The resources needed should constantly grow with the increase of the number of collected events 03/02/2010A.Minaenko3

4. Current RuTier-2 resources for ATLAS CPU, kSI2kDisc, TBATLAS Disk, TB IHEP490 13080 ITEP550 14010 JINR2150 440120 RRC-KI2000 670230 PNPI410 14060 SINP440 1503 MEPhI4206030 FIAN904623 Total6460 1730556 The total number of CPU cores is about 2500 Red – sites for user analysis of ATLAS data, the other for simulation only Now the main type of LHC grid jobs is official production jobs and CPU resources are at the moment dynamically shared by all 4 LHC VO on the base of equal access rights for each VO at each site Later, when analysis jobs will take larger part of RuTier-2 resources, the sharing of CPU resources will be proportional to VO disk share at each site 03/02/2010A.Minaenko4

5. ATLAS Storage Space ATLAS(DATA/MC)TAPE –At all T1s, write disk buffer to tape system –Centrally managed –ACL: Write permission to /atlas/Role=production Recall from tape permission to /atlas/Role=production ATLAS(DATA/MC)DISK –At all T1s and T2s, disk only pools for data and Monte Carlo –Centrally managed –ACL: write permission to /atlas/Role=production ATLASHOTDISK –At all T1s and T2s, disk only pools for hot files –Possibly enable file replication. How to do it depends on storage technology –ACL: write permission to /atlas/Role=production ATLASPRODDISK –At all T1s and T2s, buffers for Monte Carlo production –Centrally managed, automated cleanup –ACL: write permission to /atlas/Role=production 03/02/2010A.Minaenko5

6. ATLAS Storage Space ATLASGROUPDISK –At some T1s and T2s, disk only pools for Groups –Managed by the group via DDM tools. –ACL: write permission to /atlas/Role=production (!!!!) ATLASSCRATCHDISK –At all T1s and T2s, buffers user analysis data distribution datasets upload –Centrally managed, automatic cleanup 1 month dataset cache –Very subject to creation of dark data. More on this later –ACL: write permission to /atlas ATLASLOCALGROUPDISK –Non pledged space for users –Policy, ACLs, etc … defined by the site. 03/02/2010A.Minaenko6

7. Space tokens at RuTier-2 According ATLAS request all available disk space is assigned to space tokens as it is shown at the table Preliminary 3 group disks are assigned to RuTier-2 RRC-KI – exotic JINR – SM IHEP - JetEtmiss Probably we can get one more after increase of the resources At the end of 2009 CPU resources were increased by 1000 kSI2k and disk – by 1000 TB The additional resources will be available in February-March 03/02/2010A.Minaenko7

8. Estimate of resources needed to fulfil RuTier-2 tasks in 2010 The announced effective time available for physics data taking during long 2009-2010 run is 0.6*10 7 seconds ATLAS DAQ system event recording rate is 200 events per second, i.e. whole expected statistics is 1.2*10 9 events The current AOD event size is 180 KB, 1.8 times larger than ATLAS computing model requirement and hardly it will be easily decreased to the moment of the data taking Full expected size of the current AOD version is equal to 220 TB It is necessary to keep 30-40% of the previous AOD version for comparisons. This gives full AOD size of 290 TB - DATADISK During first years of LHC work the very important task is study of the detector performance characteristics and the task requires more detailed information than the one available on the AOD. ATLAS plans to use for these task “performance DPDs” which are prepared on the base of ESD. Targeted full performance DPD size is equal to full AOD size, i.e. another 290 TB - DATADISK The expected physics DPD size (official physics DPD produced by physics analysis groups) is at the level of 0.5 of full AOD size, i.e. 150 TB more - GROUPDISK 50 TB should be reserved for local users usage (ntuples, histograms kept at LOCALGROUPDISK token, 1 TB per user) - LOCALGROUPDISK Expected size of simulated AOD for MC08 (10 TeV) events only is equal to 80 TB, so need to reserve for full simulated AOD about 250 TB -MCDISK It is necessary to keep some samples of ESD and RAW events and, probably calibration data So, minimal requirement for needed disk space is at the level of 1000 TB Using usual CPU/disk ratio 3/1 one gets 3000 kSI2k estimate for needed CPU resources 03/02/2010A.Minaenko8

9. BiG Grid – the dutch e-science grid Realising an operational ICT infrastructure at the national level for scientific research (e.g. High Energy Physics, Life Sciences and others). Projects includes: hardware, operations and support. Project time: 2007 – 2011 Project budget: 29 M€ –Hardware and operations (incl. people): 16 M€ (lion’s share for HEP). 4 central facility sites ( NIKHEF, SARA, Groningen, Eindhoven ) 12 small clusters for Life Sciences wLCG Tier-1 is run as a service 03/02/2010A.Minaenko9

10. Tier-1 by BiG Grid (history) The Dutch Tier-1 (NL-T1) is run as BiG Grid service by the operational partners SARA and NIKHEF Activity initiated by PDP-group@NIKHEF (involved in EDG and EGEE) and SARA (involved in EDG and EGEE) At that point chosen for a 2 site setup –Nikhef: Compute and Disk –SARA: Compute, Disk, Mass Storage, Database and LHCOPN networking Tier-2’s connected to NL-T1, none in the Netherlands (Israel, Russia, Turkey, Ireland, Northern UK as a guest) 03/02/2010A.Minaenko10

11. T1 hardware resource for Atlas Atlas ResourcesSideDecember 2009March 2010 ComputingS14k HEPSPEC N Front End StorageS1200 TB2000 TB N1000 TB Tape StorageS800 TB2100 TB (after March) Bandwidth to tapeS450 MBps Atlas is allocated 70% of total resource for HEP 03/02/2010A.Minaenko11

12. Architecture overview 03/02/2010A.Minaenko12

13. 03/02/2010A.Minaenko Groups and group space 13

14. JINR (Dubna) MonALISA monitoring and accounting 03/02/2010A.Minaenko14

15. RuTier-2 CPU resources usage in 2009 03/02/2010A.Minaenko15

16. RuTier-2 CPU resources usage for ATLAS in 2009 ALICE – 25% ATLAS – 31% CMS – 23% LHCb – 21% JINR – 63% IHEP – 14% RRC-KI – 12% ITEP – 6% PNPI – 3% SINP – 3% 03/02/2010A.Minaenko16

17. RuTier-2 CPU resources usage in 2009 (EGEE accounting) 03/02/2010A.Minaenko17

18. NIKHEF, SARA CPU resources usage in 2009 (EGEE accounting) 03/02/2010A.Minaenko18

19. ATLAS RuTier-2 and data distribution The sites of RuTier-2 are associated with ATLAS Tier-1 SARA Now 8 sites IHEP, ITEP, JINR, RRC-KI, SINP, PNPI, MEPhI, FIAN are included in TiersOfAtlas list and FTS channels are tuned for data transfers to/from the sites 4 sites of them (IHEP, JINR, RRC-KI, PNPI) will be used by ATLAS data analysis and all physics data need for analysis will be kept at these sites. The other 4 sites will be used for MC simulations only/mostly All sites successfully participating in data transfer functional tests. This is a coherent data transfer test Tier-0 →Tiers-1→Tiers-2 for all clouds, using existing SW to generate and replicate data and to monitor data flow. Now this is a regular activity done once per week. All the data transmitted during FTs are deleted at the end of each week. The volume of the data used for functional tests is at the level of 10% of data obtained during real data taking RuTier-2 is now subscribed to get all simulated AOD, DPD, Tags as well as real 2009 AOD. The data transfer is done automatically under steering and control of central ATLAS DDM (Distributed Data Management) group The currently used shares (40%,30%,15%, 15% for RRC-KI, JINR, IHEP, PNPI) correspond to disk resources available for ATLAS at the sites MC data are transferred to MCDISK space token and cosmic and real data – to DATADISK space token at RuTier-2 Problem – during all January not a single file of reprocessed data was replicated to our sites 03/02/2010A.Minaenko19

20. STEP09 Analysis Global Results 03/02/2010A.Minaenko20

21. STEP09 lessons Data transfer part of STEP09 was successful in general. The measures have been taken to make the RRC-KI SE work more stable; the external bandwidth at IHEP has been already increased up to 1 Gbps Analysis jobs: only IHEP demonstrated good results: about 10 Hz event rate and 40% CPU efficiency. Problems with JINR were due to ATLAS sw specific bugs The main real problem: the farms in RRC-KI and PNPI needed to be reconfigured to remove bottlenecks. The farm design in the other sites need to be checked and, probably, revised also to take up future challenges. Known problems were fixed and analysis exercise repeated. The goal: try to increase frequency up to 20 Hz and CPU/wall up to 100% (see results) It is necessary to estimate output bandwidth of our separate fileservers of different types as well as full SE output bandwidth at each site for each VO. This is necessary to understand how many analysis jobs can accept each the site (see enclosed presentation) It is not done yet. Will be done after the resources increase. The problem is that ratio (Output bandwidth)/(Fileserver volume) is different for different fileservers due to historical reasons. This leads to the fact that full SE bandwidth can be considerably less than just a sum of fileserver bandwidths. It is hard to fix the problem but it is just necessary to take this into account for future equipment purchases. 03/02/2010A.Minaenko21

22. Hammercloud test #895 03/02/2010A.Minaenko22

23. Hammercloud test #895 03/02/2010A.Minaenko23