1. Tony Doyle a.doyle@physics.gla.ac.uk Status Report, GridPP6 Collab. Meeting, Coseners House, 31 January 2003

2. Tony Doyle - University of GlasgowOutline Historical Perspective? Prototype to Production The case so far… Timelines… Presentation to PPARC Science Committee (26/11/02): Questions from Science Committee Motivation Overview How Does the Grid Work? Middleware Development Testbed Status Is the Middleware Robust? LHC Computing Grid Applications Tier-1 and -2 Centre Resources Dissemination Achievements Timeline UK Grid Future Priorities Roadmap: Not a Bid Preliminary Input

3. Tony Doyle - University of GlasgowOutline Questions from Science Committee Motivation Overview How Does the Grid Work? Middleware Development Testbed Status Is the Middleware Robust? LHC Computing Grid Applications Tier-1 and -2 Centre Resources Dissemination Achievements Timeline UK Grid Future Priorities Roadmap: Not a Bid Preliminary Input

4. Tony Doyle - University of Glasgow Questions from Science Committee 1.What has been achieved in the first year? See Oversight Committee (3): Executive Summary 2.How are the finances being spent? Broken down into 5 areas 3.Development in the medium-long term and resources required? GridPP I (09/01-08/04)Prototypeshort GridPP II (09/04-08/07)Productionmedium (09/07-) Exploitationlong-term

5. Tony Doyle - University of Glasgow Rare Phenomena – Huge Background 9 orders of magnitude The HIGGS All interactions

6. Tony Doyle - University of Glasgow Executive Summary Introduction Project Management Resources CERN DataGrid Applications Tier-1/A Tier-2 Dissemination Future Funding Ref: PMB-13-EXEC the Grid is starting to work under control via Project Map Small modifications Manpower OK. Hardware Making impact Engaged (value added) Tier-A production mode Latent resources UK flagship project Preliminary planning

7. Tony Doyle - University of Glasgow Overview EDG - UK Contributions Architecture Testbed-1 Network Monitoring Certificates & Security Storage Element R-GMA LCFG MDS deployment GridSite SlashGrid Spitfire… Applications (start-up phase) BaBar CDF/D0 (SAM) ATLAS/LHCb CMS (ALICE) UKQCD £17m 3-year project funded by PPARC through the e-Science Programme CERN - LCG (start-up phase) funding for staff and hardware... CERN DataGrid Tier - 1/A Applications Operations http://www.gridpp.ac.uk

8. Tony Doyle - University of Glasgow £17m++ 3-Year Project Five components –Tier-1/A = Hardware + CLRC e-Science Staff –DataGrid = 15 DataGrid Posts + CLRC PPD Staff –Applications = 13 Experiments Posts (to interface middleware) –Operations = Travel (~100 people)+ Management + Early Investment –CERN = 25 LCG posts + Tier-0 + LTA

9. Tony Doyle - University of Glasgow Project Management - 7 Elements

10. Tony Doyle - University of Glasgow Year 0 Year 1 The Project has now completed one year ….

11. Tony Doyle - University of Glasgow How Does the Grid Work? 1. Authentication grid-proxy-init 2. Job submission dg-job-submit 3. Monitoring and control dg-job-status dg-job-cancel dg-job-get-output 4. Data publication and replication globus-url-copy, GDMP 5. Resource scheduling – use of Mass Storage Systems JDL, sandboxes, storage elements 0. Web User Interface…

12. Tony Doyle - University of Glasgow Middleware Development

13. Tony Doyle - University of Glasgow Testbed Status

14. Tony Doyle - University of Glasgow Is the Middleware Robust? 1.Code Base 2.Software Evaluation Process 3.Testbed Infrastructure: WP-specific Development Certification Application. 4.Code Development Platforms

15. Tony Doyle - University of Glasgow LHC Computing Grid Grid deployment project Not grid development Establishes the global computing infrastructure Allows all participating physicists to exploit LHC data Fosters and develops the required collaboration between –LHC experiments –peer computing centres –middleware providers Based at CERN Started March 2002

16. Tony Doyle - University of Glasgow LHC Computing Grid Solid programme established Agreed between and supported by all LHC experiments LHC experiments contributing to development of common software (e.g. new persistency solution) First LHC global grid service for deployment in July 2003 Basis for a generic Science Grid Infrastructure

17. Tony Doyle - University of Glasgow Application Interfaces

18. Tony Doyle - University of Glasgow Tier-1/A: Year of Growth GridPP Certificates BaBar Use

19. Tony Doyle - University of Glasgow Tier-2 Web-Based Monitoring ScotGrid reached its 100,000th processing hour on Wednesday 13th November 2002.

20. Tony Doyle - University of Glasgow Tier-1 and -2 Centre Resources Estimated resources at end of 2003 (from Institute returns) Number of CPUs Amount of Disk (TB) Amount of Tape (TB) London1252 76 8 Northern201511333 Southern 392 2920 ScotGrid 154 35 0 Number of CPUs Amount of Disk (TB) BaBar110829%7630% CDF45312%3915% D03208% 83% ATLAS94725%7228% CMS3048% 94% LHCb54314%4016% Tier-1 Tier-1: 600 CPUs + 150 TB Tier-2: (4000 CPUs + 200 TB)

21. Tony Doyle - University of Glasgow Dissemination: e-Science and Web e-Science All Hands Meeting held at Sheffield, 2-4 September 2002 –~ 300 people in total –~ 19 GridPP People attended –~ 13 GridPP Abstracts accepted (total ~100) –~ 10 GridPP Posters displayed – 4 GridPP Invited talks – 3 GridPP Demonstrations GridPP Web Page Requests: 25,000 per month ~ ½ non-UK.com (commercial)

22. Tony Doyle - University of Glasgow Dissemination: Posters ATLASSAM OptorSimGridPPTier-1/AScotGrid BaBarLHCbCMS Storage 10 Posters for NeSC Opening and e-Science All Hands Meeting

23. Tony Doyle - University of Glasgow Dissemination: Demonstrations Super Computing 2002 Baltimore, US Major event last week GridPP participated in three successful Worldwide demos –WorldGrid –Replica Location Service –SAMGrid These and other Web-based demos available online from http://www.gridpp.ac.uk/demos/

24. Tony Doyle - University of Glasgow Achievements I 1.Dedicated people actively developing a Grid 2.All with personal certificates 3.Using the largest UK grid testbed (16 sites and more than 100 servers) 4.Deployed within EU-wide programme 5.Linked to Worldwide Grid testbeds

25. Tony Doyle - University of Glasgow Achievements II 6.Grid Deployment Programme Defined The Basis for LHC Computing 7.Active Tier-1/A Production Centre meeting International Requirements 8.Latent Tier-2 resources being monitored 9.Significant middleware development programme 10.First simple applications using the Grid testbed (open approach)

26. Tony Doyle - University of Glasgow GridPP From Prototype (hundreds) to Production (thousands) Grid.. 2002200520042003 2001 General deployment of e-Science methods… Certs.

27. Tony Doyle - University of Glasgow Timeline Timeline 2002200520042003 Q1 Q2 Q3 Q4 GridPP- Procure, Install, Compute, Data Develop, Test, Refine LHC Computing Grid Initial Grid Tests EGEE (DataGrid II?) GridPP II Grid Service Prototypes Production 2001 Q1 Q2 Q3 Q4 DataGrid Middleware and Hardware upgrades Worldwide Grid Demonstrations Transition and Planning Phase…

28. Tony Doyle - University of Glasgow UK Grid Future Priorities 1.Tier-1/A staff 2.Tier-1/A hardware 3.Tier-2 staff 4.Applications staff 5.Middleware development 6.Tier-2 hardware 7.CERN staff 8.CERN hardware Preliminary assessment phase… ALL of these are required to address the LHC Computing Challenge

29. Tony Doyle - University of Glasgow Roadmap: Not a Bid Preliminary Input GridPP IGridPP IIExploitation GridPP II era

30. Tony Doyle - University of Glasgow UK Grid Future Priorities 1.Tier-1/A staff – National Grid Centre 2.Tier-1/A hardware – International Role 3.Tier-2 staff – UK e-Science Grid 4.Applications 1.Grid-enablement 2.Applications development 5.Middleware – EU-wide development 6.Tier-2 hardware – non-PPARC funding 7.CERN staff – UK pro-rata contribution 8.CERN hardware – pro-rata contribution Preliminary assessment phase… ALL of these are required to address the LHC Computing Challenge

31. Tony Doyle - University of GlasgowHistory Teaches us nothing? Review latest status There was ~nothing 18 months ago…

32. Tony Doyle - University of Glasgow Application Testbed Resources Since Last Year: –Improved software (EDG 1.4.3). –Doubled sites. More waiting… Australia, Taiwan, USA (U. Wisc.), UK Sites, CrossGrid, … –Significantly more CPU/Storage. Hidden Infrastructure –MDS Hierarchy –Resource Brokers –User Interfaces –VO Replica Catalogs –VO Membership Servers –Certification Authorities SiteCountryCPUsStorage CC-IN2P3*FR620192 GB CERN*CH1381321 GB CNAF*IT481300 GB Ecole Poly.FR6220 GB Imperial Coll.UK92450 GB LiverpoolUK210 GB ManchesterUK915 GB NIKHEF*NL142433 GB OxfordUK130 GB PadovaIT11666 GB RAL*UK6332 GB SARANL0254 GB TOTAL510755223 GB

33. Tony Doyle - University of GlasgowHistory VersionDate 1.1.227Feb2002 1.1.302Apr2002 1.1.404Apr2002 1.2.a111Apr2002 1.2.b131May2002 1.2.012Aug2002 1.2.104Sep2002 1.2.209Sep2002 1.2.325Oct2002 1.3.008Nov2002 1.3.119Nov2002 1.3.220Nov2002 1.3.321Nov2002 1.3.425Nov2002 1.4.006Dec2002 1.4.107Jan2003 1.4.209Jan2003 1.4.314Jan2003 RC Changes Mixed Globus 2.0/2.2 RB/JSS Upgrade Known Problems: GASS Cache Coherency Race Conditions in Gatekeeper Unstable MDS Successes Improved MDS Stability FTP Transfers OK Known Problems: Interactions with RC Intense Use by Applications! Limitations: Resource Exhaustion Size of Logical Collections Successes Matchmaking/Job Mgt. Basic Data Mgt. Known Problems: High Rate Submissions Long FTP Transfers

34. Tony Doyle - University of Glasgow History -relating applications work to TB versions VersionDate 1.1.227Feb2002 1.1.302Apr2002 1.1.404Apr2002 1.2.a111Apr2002 1.2.b131May2002 1.2.012Aug2002 1.2.104Sep2002 1.2.209Sep2002 1.2.325Oct2002 1.3.008Nov2002 1.3.119Nov2002 1.3.220Nov2002 1.3.321Nov2002 1.3.425Nov2002 1.4.006Dec2002 1.4.107Jan2003 1.4.209Jan2003 1.4.314Jan2003 RC Changes Mixed Globus 2.0/2.2 RB/JSS Upgrade Known Problems: GASS Cache Coherency Race Conditions in Gatekeeper Unstable MDS Successes Improved MDS Stability FTP Transfers OK Known Problems: Interactions with RC Real Use by Applications! Limitations: Resource Exhaustion Size of Logical Collections Successes Matchmaking/Job Mgt. Basic Data Mgt. Known Problems: High Rate Submissions Long FTP Transfers ATLAS commence phase1 tests CMS start stress tests Nov 30 which continue till Dec 20 Problems with long jobs Instability in MDS Long file transfers unreliable CMS and ATLAS evaluate 1.4.3

35. Tony Doyle - University of Glasgow EDG reasons of failure (categories) Preliminary analysis Short jobs Long jobs

36. Tony Doyle - University of Glasgow EDG Release Plan on Application Testbed Port to RH 7.3 & LCFGng January Upgrade to globus 2.2.4 & condor 6.4.6 February R-GMA RLS Storage Element NetworkCost Suite Replica Mgmt Services March & April LCAS with dynamic plug-in modules New Resource Broker + GLUE schema GridFTP access to CASTOR VOMS https://edms.cern.ch/document/333297 LCG defines components for first release (not just EDG) Deployed May 2003

37. Tony Doyle - University of Glasgow The Road Ahead? Plan for success in middleware development … http://www.roadahead.com/ First two links are broken.. Define the Production Phase as GridPP II (Sep 04…)

38. Tony Doyle - University of Glasgow GridPP Timeline… 29/10/02CB ~outline approval for GridPP2 (Prototype to Production) and EGEE (Grid = Network++) Dec 02EU Call for FP6 (EGEE) Jan 03GridPP Tier-2 tenders Four ½ posts to be assigned to regional management (Testbed: Grid enablement of Hardware) Feb 03PPARC Call for e-Science proposals 19/02/03CB review of GridPP II plans Apr 03EGEE Proposal submission ** Tier-1 and Tier-2 Centres not yet defined (work with prototypes)** End-May 03GridPP II Proposal Sep 03Approval? (Limited) Tier-2 funding starts Dec 03DataGrid funding ends ~Jan 04DataGrid as part of EGEE? Sep 04Start of GridPP II Production Phase…