1. LCG-France: les Tier-2s et Tier-3s 2 ème Colloque LCG-France Clermont-Ferrand, 14 mars 2007 Tier-2: Subatech Tier-2: LPC Tier-2: GRIF CEA/DAPNIA LAL LLR LPNHE IPNO Tier-2: GRIF CEA/DAPNIA LAL LLR LPNHE IPNO AF: CC-IN2P3 Tier-3: LAPP Tier-1: CC-IN2P3 Tier-3: IPHC Lyon Clermont-Ferrand Ile de France Marseille Nantes Strasbourg Annecy Tier-3: IPNL Tier-3: CPPM

2. F. Chollet/F. Hernandez 2 Table des Matières Introduction  Computing model, Rôle des Tiers-2, Services de base Sites LCG-France  Contribution des Tier-2s et Tier-3s Groupe technique LCG-France T2-T3  Activités des Tier-2s et Tier-3s Participation aux Service Challenge LCG et aux exercices expérimentaux  Quelques résultats Perspectives 2007

3. F. Chollet/F. Hernandez 3 Introduction Needs from LHC experiments : Computing facilities  Storage capacities –Collisions: ~10 PB/year Recoding rate up to 1 Gb/s –Simulation of large sets of Monte Carlo data  CPU –Processing and reprocessing –Monte Carlo Simulation and reconstruction –Analysis  Inter-sites large bandwidth –Distribution of large volume of scientific data worldwide : from CERN to the computing sites and from site to site for the simulated data Data handling and Computing problem : Computing model  Distribution of Tasks  Distribution of Data among a hierarchy of distributed sites

4. F. Chollet/F. Hernandez reconstruction analysis simulation interactive physics analysis batch physics analysis batch physics analysis detector event summary data raw data event reprocessing event reprocessing event simulation event simulation analysis objects (extracted by physics topic) Data Handling and Computation event filter (selection & reconstruction) event filter (selection & reconstruction) processed data les.robertson@cern.ch Tier-2 Tier-1 Tier-0

5. F. Chollet/F. Hernandez 5 Tasks Distribution Tier-2  Simulation Monte-Carlo  Interactive and batch end-user data analysis More than 40 % of the required CPU in Tiers-2, outside CERN and Tiers-1 Difference between T1 and T2 for ALICE is only QoS Experiment computing models define share between analysis and simulation depends  Only simulation for LHCb  Both for ATLAS and CMS Tier-3  Used by experiments as available  End-user analysis  Code development, testing, ntupling … Initiative sans engagement formalisé vis-à-vis de LCG Tier-0 (1) Tier-1 (11) Tier-2 (~120) Tier-3 Tier-1s and Tier-2s are optimized for computing model efficiency and Tier-3s for physicist efficiency

6. F. Chollet/F. Hernandez 6 Tier-1 Tier-2 Data Distribution Inter-sites relationships Experiment computing models define specific data flows between sites and inter-sites relationships quantifying the storage and network services required at each level RAW data distribution from Tier 0 to Tiers-1 viewed as online for the data acquisition process Simulated data produced in Tiers-2 uploaded to the associated Tier-1 Number of Tiers-2 and even Tiers-3 still growing : LHC Computing will be more dynamic and network oriented For Tiers-2, relation with the corresponding T1 is fundamental

7. F. Chollet/F. Hernandez 7 Baselines Services in Tiers-2 Provisioning a managed grid-enable disk storage  permanent and temporary data storage  No MSS Provisioning a access to the stored data by other centre of the grid  Any Tier 2 may have access data to any Tier 1  In particular, the network bandwidth and services for data exchange with associate Tier 1 should be ensure. Provisioning a processing capacity  Should be access by the grid standards Provisioning some services according to the experiment requirements  Collective or Dedicated to experiences grid services (VOBox,…)

8. F. Chollet/F. Hernandez 8 Sites LCG-France (1/3) WLCG Tiers-2 : Analysis facility in Lyon and 3 Tiers-2 declared:  GRIF in Paris Region acting as a federation of 5 sites : DAPNIA, IPNO, LAL, LLR, LPNHE  LPC Clermont  Subatech Nantes Memorandum of Understanding agreed in October 2005, now being signed  Qualification criteria for inclusion not just funding and threshold size “ As a guideline, Tiers-2 are each expected to provide at least a few percents of the resources required by the LHC experiments they serve”  Long term commitment in terms of service level and response times  Average availability measured on annual basis need to be upper at 95 % ( 18 days per year of unavailability)  Responding delay to operational problems : from 2 hours to 72 depending of the services affected and the periods where the problem occur.

9. F. Chollet/F. Hernandez 9 Sites LCG-France (2/3) LCG-France : a set of 3 Tier-2s and 4 Tier-3s  Tiers-3 : CPPM, IPHC, IPNL, LAPP  GRIF and Analysis facility in Lyon splitted  W-LCG Tier-2 : 80 % of GRIF resources  LCG-France Tier-3 : 20 % of GRIF resources  1 more Tiers-3 candidate (LPSC Grenoble)  All sites also support non LHC virtual organizations  Tier-3 strategy : small Tier-2 and local Analysis facility Tier-2: Subatech Tier-2: LPC Tier-2: GRIF CEA/DAPNIA LAL LLR LPNHE IPNO Tier-2: GRIF CEA/DAPNIA LAL LLR LPNHE IPNO AF: CC-IN2P3 Tier-3: LAPP Tier-1: CC-IN2P3 Tier-3: IPHC Lyon Clermont-Ferrand Ile de France Marseille Nantes Strasbourg Annecy Tier-3: IPNL Tier-3: CPPM

10. F. Chollet/F. Hernandez 10 Sites LCG-France (3/3) Scientific and Technical project leaders : CC-IN2P3 Lyon : F.Malek, F. Hernandez GRIF Grille de Recherche Ile de France : JP. Meyer, M. Jouvin LPC Clermont-Ferrand : D. Pallin, J.C Chevaleyre Subatech Nantes : L. Aphecetche, JM. Barbet CPPM Marseille : C.Bee,T.Mouthuy LAPP Annecy : S.Jézéquel, N.Neyroud IPHC Strasbourg : D.Bloch,Y.Patois IPNL Lyon : S.Perries, D.Pugnère Technical Teams : : G.Baulieu,JM. Barbet, J.Bernier, D.Bouvet, B.Boutherin, L.Caillat, K.Chawoshi, C.Diarra,S. Elles, E. Fede, Y.Giraud, P.Girard, M. Gougerot, C. Gondrand, E. Knoops, P. Micout, P. Larrieu, C. Leroy, C. L’Horphelin, L. Martin, E. Medernach, T.Ollivier, Y.Perret, G.Philippon, G. Rahal, M. Ricard, R. Rumler, F. Schaer, J. Schaeffer, L. Schwarz, I. Semeniouk, D. Terront, A. Trunov

11. F. Chollet/F. Hernandez 11 Contribution Tier-2s (1/3) Computing resources in 2008  WLCG : ~ 40 % of the total CPU resources are required in the Tier-2s  Analysis Facility and Tier-2s : 33 % of the total CPU resources pledged by LCG-France No revision of LCG-France Tier-2s planned capacity has been done yet according to new estimates of computing capacity requirements (october 2006) CPU [k SI2000]Storage [TB] Number of average Tier-2 Number of Tiers-2 Average Required Capacity per T2 in 2008 LCG-France Tiers-2 in 2008 Average Required Capacity per T2 in 2008 LCG-France Tiers-2 in 2008 Alice 600740150175 1,216 Atlas 7001790310720 2,625 CMS 5501060150330 2,028 LHCb 400252060 0,611

12. F. Chollet/F. Hernandez 12 Tier-2s Contribution (2/3) Number of Tier-2 sites registered : Alice: 16 Atlas: 25 CMS: 28 LHCb: 11 Number of Tier-2 sites registered : Alice: 16 Atlas: 25 CMS: 28 LHCb: 11 Do not include yet any revision of planned capacity according to new estimates of computing capacity requirements. Do not include yet any revision of planned capacity according to new estimates of computing capacity requirements.

13. F. Chollet/F. Hernandez 13 Tier-2s Contribution (3/3) Number of Tier-2 sites included : Alice: 16 Atlas: 25 CMS: 28 LHCb: 11 Number of Tier-2 sites included : Alice: 16 Atlas: 25 CMS: 28 LHCb: 11

14. F. Chollet/F. Hernandez 14 Tier-2s Planned Capacity in 2008 2.5 M SpecInt2000 : 12/2006 CPU capacity of CC-IN2P3 2.5 M SpecInt2000 : 12/2006 CPU capacity of CC-IN2P3 1000 kSI2000 ≈ 320 CPUs (AMD Opteron 275 dual-core ) Demande ATLAS: Partie non MC de l’AF de Lyon considérée comme un T3

15. F. Chollet/F. Hernandez 15 Tier-2s Planned Evolution 43% of the planned Tier-1 disk capacity the same year Roughly equivalent to the planned Tier-1 CPU capacity the same year

16. F. Chollet/F. Hernandez 16 Tier-3s Planned Evolution 1000 kSI2000 ≈ 320 CPUs (AMD Opteron 275 dual-core) To be checked with Tier-3s : overall resources versus LCG resources

17. F. Chollet/F. Hernandez 17 LCG-France T2-T3 Technical group  set up in April 2006 – Coordination 2 years mandate  mailing list, regular visio conferences, wiki pages Activities  Network-level and SRM-level data transfer tests from and to tier-1  Including associated foreign sites  Meetings held with several potential hardware providers  Sharing of technical and commercial information (hardware evaluation results, commercial conditions, etc.)  Quattor expertise  GRIF very active in the Quattor working group  Used by most of the LCG-France sites

18. F. Chollet/F. Hernandez 18 LCG-France T2-T3 Technical group In close contact with some foreign associated tier-2s  Europe  Belgium CMS Tier-2  Romanian Federation ATLAS Tier-2  Asia  IHEP China - ATLAS and CMS Tier2  ICEPP Japan - ATLAS Tier2

19. F. Chollet/F. Hernandez 19  With EGEE SA1: Grid Operations  Baseline services integration  Central services deployment in Lyon Tier-1  Metrics estimate : Site availability  Infrastructure support and monitoring  With Experiments : LHC computing tracking  Capacity requirements, Connectivity bandwidth estimates  Collaboration with experimental representatives  being part of the Service Challenges and experiments driven tests  exercise the reliability of the models of data distribution and processing  Task force “Efficacité de production)  With Network experts in IN2P3 and Renater NREN Collaborative work

20. F. Chollet/F. Hernandez 20 Network bandwith estimates on-going work Tier-1 and Tiers-2 connectivity relies on NREN Very first estimates form WLCG for 2008 in December 2006 Experiments needs : 1 MB/s -> Network bandwith : 20 Mb/s Tiers-2 estimates GRIF-CC + GRIF internal connexions to be studied J.Bernier CCIN2P3 T2inout LPC 60 Mb/s140 Mb/s Subatech 60 Mb/s GRIF 1060 Mb/s650 Mb/s

21. F. Chollet/F. Hernandez 21 Network bandwith estimates on-going work CCIN2P3 in outMb/s CERN T03200 100-> 10Gb/s directe Autres T150005300-> + 10Gb/s FZK France T2 850 880-> backbone RENATER Autres T2 6003500-> connexion directe CC-GEANT ? Support of foreign Tiers-2 required to be studied in terms of network connectivity

22. F. Chollet/F. Hernandez 22 T2-T3 Site availability Measuring site availability by using SAM Source: http://lcg-sam.cern.ch:8080/sqldb/site_avail.xsqlhttp://lcg-sam.cern.ch:8080/sqldb/site_avail.xsql Courtesy of Frédérique Chollet

23. F. Chollet/F. Hernandez 23 SC4 Service Challenge Service Challenge : real grid service  Understand what it takes to operate a real grid service  Verify Tier-0, Tiers-1, Tiers-2 deployment with realistic usage patterns Pilot LHC Service from June to September 2006 A stable service on which experiments can make a full demonstration of experiment offline chain  DAQ  Tier-0  Tier-1 : data recording, calibration, reconstruction  Offline analysis - Tier-1  Tier-2 data exchange simulation, batch and end-user analysis Extension to most Tier-2 sites  Data transfers from T1s and T2s  Stable transfers over large periods of time  Keep transfers at the rates expected for data distribution  MC production and data replication And sites can test their operational readiness  Service metrics  MoU service levels  Grid services

24. F. Chollet/F. Hernandez 24 ATLAS Production at Tier-2s

25. F. Chollet/F. Hernandez 25 Atlas SC4 Service Challenge ATLAS: Dataset transfer exercise from T1 to T2-T3  Simultaneous transfers to the 7 T2-T3 sites for more than 24 hours; it reached more than 25MB/s (27/07/06)  DDM Monitoring : http://atlas-ddm-monitoring.web.cern.ch/atlas- ddm-monitoring/ http://atlas-ddm-monitoring.web.cern.ch/atlas- ddm-monitoring/

26. F. Chollet/F. Hernandez 26 Alice PDC’06 Total amount of CPU provided by T2s Support provided in sites : Artem Trunov (CCIN2P3), Jean-Michel Barbet (Subatech) Subatech: 14% CC+T2s 63% T2s GRIF: 5% CC+T2s 23% T2s LPC: 3% CC+T2s 14% T2s CC: 78% CC+T2s Monitoring : http://pcalimonitor.cern.ch:8889http://pcalimonitor.cern.ch:8889

27. F. Chollet/F. Hernandez 27 CMS CSA06 exercise Computing, Software, Analysis challenge of 2006  Data transfers exercise : T1 receive data at rate prop. to 25 %  Modified hierachical model : Data access may come from a variety of T1s Results for GRIF above the 5MB/s threshold for sucess

28. F. Chollet/F. Hernandez 28 LHCb DC’06 Monte Carlo production http://lhcb.pic.es/DIRAC/Monitoring/Production / Angelo Carbone Raja Nandakumar (DC06 Coordinators) Used Resources from 2006-05-10 to 2006-09-07 By Country Total number of events produced 144M SiteEvents (%)Events UK29.0041,777,300 CH19.7528,437,622 IT15.5722,429,760 FR10.2814,799,830 ES6.9910,065,229 GR5.247,552,039 DE5.007,206,019 NL2.894,156,350 PL1.802,584,749 RU1.602,305,570 BG0.771,108,510

29. F. Chollet/F. Hernandez 29 Monte Carlo production http://lhcb.pic.es/DIRAC/Monitoring/Production / Angelo Carbone Raja Nandakumar (DC06 Coordinators) Total number of events produced 144M over ~100 sites Used Resources from 2006-05-10 to 2006-09-07 By Site LHCb DC’06 SiteEvents (%)Events LCG.CERN.ch18.5926,773,752 LCG.CNAF.it11.1816,105,760 LCG.RAL.uk10.6015,264,775 LCG.LPC.fr5.818,372,270 LCG.Manchester.uk4.766,853,250 LCG.QMUL.uk4.696,754,999 LCG.GRIDKA.de4.105,910,519 LCG.USC.es3.074,420,500 LCG.NIKHEF.nl2.824,055,101 DIRAC.Lyon.fr2.643,807,749 LCG.Barcelona.es2.373,417,810 LCG.Liverpool.uk2.273,269,500

30. F. Chollet/F. Hernandez 30 Plans for 2007 Storage space provision is a major concern for all Tiers –Data access patterns required by the expriments –Managed disk enabled storage : SRM v2.2 implemetation –File Systems studies (GPFS, Lustre evaluation) and GSI enabled protocols VOMS groups and roles integration Site availability : improve stability despite on-going activities at sites –Infrastructure consolidation, hardware procurements, OS evolution, Mware upgrade –Electric and cooling infrastructure is an issue –Running over XMas, holidays period… Efficiency : Plans for a close collaboration with the new TFEP (Task Force Efficacité de Production) –Improve the global efficiency of ATLAS production on the FR cloud More on Grid Security in connection with IN2P3 security managers (B.Boutherin, B. Delaunay)

31. F. Chollet/F. Hernandez 31 Proposition du Management LCG-France Début de la réflexion en 2007 Termes de cet accord ?….. Préliminaire….  Engagement des Tier-2s à satisfaire aux critères W-LCG et à pourvoir aux moyens matériel et aux ressources humaines pour la durée du LHC  Engagement d’entraide mutuelle entre le Tier-1 et les Tier-2s  Engagement de l’IN2P3/Dapnia pour le maintien des ressources financières à partir de 2009 permettant le renouvellement par tiers et par an du matériel et la mise à disposition (ou embauche) de personnel qualifié pour faire tourner les centres sous conditions (demandes communes et mise en place d’une procédure d’arbitrage : commission ad-hoc, examen en comité de pilotage….)  Examen des Tier-2s en comité de pilotage 1 fois l’an Protocole d’accord IN2P3/DAPNIA - T2s français

32. F. Chollet/F. Hernandez 32 Conclusions Additional resources coming from Tier-2s & even Tier-3 initiatives  Not in competition with Tier-1 funding but funding support expected in 2009  Significant effort in terms of Budget, infrastructure, human, technology… Collaborative work  Within EGEE SA1  Resources and base line services  Within LCG-France  Tier-1 – Tier-2s Tier-3s integration  Enhance collaboration between sites experts and experiment representatives  Relation with the corresponding T1 is fundamental  Working together with experiences  Experiment computing models define tasks distribution, data distribution and specific data flows between Tier-1s and Tier-2s

33. F. Chollet/F. Hernandez 33 Documents de référence W-LCG Reference documents & Source :  MoU http://lcg.web.cern.ch/LCG/C-RRB/MoU/WLCGMoU.pdf  Summary of Regional Centres Capacity 10/10/06 http://lcg.web.cern.ch/LCG/planning/phase2_resources/ http://lcg.web.cern.ch/LCG/planning/phase2_resources/  Revised Computing Capacity Requirements October 2006 / previous TDR estimates http://lcg.web.cern.ch/LCG/MB/revised_resources/  New estimates Comparaison of New requirements with Current pledges : http://lcg.web.cern.ch/LCG/MB/revised_resources/NewRequOldPledge s241006.pdf http://lcg.web.cern.ch/LCG/MB/revised_resources/NewRequOldPledge s241006.pdf LCG-France Reference documents :  LCG-France Tier-2 Tier-3 Resource Planning 2006-2010 https://edms.in2p3.fr/document/I-008142/