1. 1 GANGA, LHCb and ATLAS Planning Glenn Patrick GRIDPP11 – 14 September 2004

2. 2 ATLAS/LHCb GANGA Planning User Grid Interface. Both experiments already use common GAUDI/ATHENA framework. Different analysis projects (DaVinci, DIAL). LHCb ATLAS BaBar…

3. 3 GANGA User Interface Database of Standard Job Options Job Options Editor Strategy Database (Splitting scripts) Strategy SelectionData Selection (Input/Output Files) Job Requirements (LSF Resources, etc) Job Factory (Job Registry Class) Ganga Job object Local Client Grid/Batch System Gatekeeper Submit job Send job output Worker nodes Get job output Send Job script JDL file Job Options file Get Monitoring Info Storage Element File Transfer

4. 4 User actions to submit an analysis job DaVinci Workflow Select WorkFlow Prepare Sandbox Prepare AlgFlowOptions and DLLs DaVinci AlgorithmFlow Edit AlgorithmFlow AlgParamOptions Select Datasets Edit AlgParamOptions DatasetOptions AlgFlowOptions Sandbox DLLs JobOptions FileCatalog slice Submit Job Metadata catalog AlgOptions catalog DLLs File catalog Select available version Select available top options file Edit algorithm workflow and parameters of algoritms Select input datasets Press Submit button Watch job status See job output in the job folder Check out DaVinci, prepare and compile your packages Outside of Ganga JOE Bookkeeping DB interface Using Ganga Without Ganga

5. 5 GANGA – ATLAS/LHCb GRIDPP1 (Karl Harrison, Alexander Soroko + A.Tan, J.Martyniak) delivered a prototype in April 2003. Versions being deployed for 2004 Data Challenges. For GRIDPP2, develop production version capable of submitting full range of ATLAS/LHCb applications to various Grids. Support complex analysis environments. 7Level-1 deliverables. 25Level-2 deliverables. Level-1 deliverables relate to releases of increasing functionality leading up to real data. Large agreement between the two experiments. Still some fine tuning.

6. 6 GANGA – ATLAS & LHCb Name:Grid front-end for ATLAS analysis tasks, end-to-end analysis solution for LHCb. Why:Shield the user from Grid middleware and provide an easy way to change between different Grid implementations. Client:ATLAS, LHCb Achievement:Effective and significant use by ATLAS and LHCb user communities. High level Risks: Contention between ATLAS and LHCb objectives. Low take-up because of competition from ad hoc non- generic solutions. Sudden changes in experiment frameworks. Delays in middleware development. Delays in metadata project.

7. 7 GANGA: Level-1 Deliverables Deliverable G1 (end Feb 2005) Optimise GANGA design for compatibility with other relevant systems. (Metric: software release) Deliverable G2 (end April 2005) Release with full support for plug-in modules. (Metric: software release and associated documentation) Deliverable G3 (end October 2005) Release with full job-building capabilities. (Metric: software release and associated documentation) Deliverable G4 (end March 2006) Release with support for full range of ATLAS and LHCb applications. (Metric: software release and associated documentation) Deliverable G5 (end November 2006) Release that supports all available functionality on the full range of Grid/batch back-ends. (Metric: software release and associated documentation) Deliverable G6 (end April 2007) Release that maximises Ganga portability and allows for user mobility. Deliverable G7 (end September 2007) Physics optimised release of GANGA. Metric: software release and associated documentation)

8. 8 LHCb-specific GridPP Development LHCb development is on three fronts: Production Control and Monitoring Gennady Kuznetsov (RAL) Data Management Carmine Cioffi (Oxford) GANGA ( with ATLAS) Alexander Soroko (Oxford) Karl Harrison (Cambridge) All developed in tandem with LHCb Data Challenges UK Contribution DC2003 34% events DC2004 26% events

9. 9 DIRAC Services and Resources DIRAC Job Management Service DIRAC Job Management Service DIRAC CE LCG Resource Broker Resource Broker CE 1 DIRAC Sites Agent CE 2 CE 3 Production manager Production manager GANGA UI User CLI JobMonitorSvc JobAccountingSvc AccountingDB Job monitor InfomarionSvc FileCatalogSvc MonitoringSvc BookkeepingSvc BK query webpage BK query webpage FileCatalog browser FileCatalog browser User interfaces DIRAC services DIRAC resources DIRAC Storage DiskFile gridftp bbftp rfio

10. 10 LHCb Production Control GRIDPP1 (Gennady Kuznetsov) provided the Production Desktop for building complex workflows from components – DC04. For GRIDPP2, a Production Console is being written to support/configure the production environment for the Production Manager. 6Level-1 deliverables. 17Level-2 deliverables. Dependent on Data Challenge schedule, LCG releases, etc. 2 GRIDPP1 deliverables included. All deliverables well established. Agreed with post-holder. Step Workflow Definition

11. 11 LHCb Production Control DIRAC Objective: Production Control and Monitoring Name:Production Control and Monitoring Why:Provide environment for Production Manager to control and monitor LHCb production. Client:LHCb Collaboration Achievement:Provision of effective central control of LHCb production so that system can be run by single person. High level Risks: Undermining by changing underlying technology (Grid deployments, etc). Loss of expertise or delays due to possibility of illness/relocation of single developer. Radical changes in requirements due to fundamental change in philosophy of distributed production system and data model.

12. 12 LHCb Production Control : Level-1 Deliverables Deliverable LP1 (2004, end Q4) Integration of Production Desktop with DIRAC. (Metric: software release. demonstration) Deliverable LP2 (2005 end Q1) Requirements and Architecture of the Production Console. (Metric: documents) Deliverable LP3 (2005 end Q2) Release 0 (first prototype) of the Production Console for 2005 Data Challenge. (Metric: software release) Deliverable LP4 (2006 end Q1) Release 1 for 2006 Data Challenge with capability for continuous production rather than periodic data challenges. (Metric: software release, demonstration) Deliverable LP5 (2006 end Q4) Release 2 of the Production Console with data management functions and to support processing of real data. (Metric: software release, demonstration) Deliverable LP6 (2007 end Q3) Final release based on production experience of real collisions (if available). (Metric: software release, demonstration)

13. 13 LHCb Data Management GRIDPP1 (Carmine Cioffi) contributed to the POOL File Catalogue Browser and integration of the POOL persistency framework into Gaudi (new EventSelector interface). Focus for GridPP2 is on metadata and the deployment of a new Metadata Catalogue Service. Optimised for LHCb, but work with common ARDA and UK metadata projects (web-service technology, ARDA test-suite). 9Level-1 draft deliverables. 23Level-2 draft deliverables. This post had no formal GRIDPP1 milestones. Need to also align with deliverables of UK metadata posts. Still iterating - under discussion this week (LHCb s/w week). Dataset Event 1 Event 2 … Event 3 Dataset Event 1 Event 2 … Event 3 File Event 1 Event 2 … Event N

14. 14 LHCb Data Management Objective: Data Management Name:Data Management Why:A dynamically optimised and high availability database service is required to provide a metadata and job provenance catalogue for the experiment. This will provide efficient storage and query mechanisms to the experiment for metadata & job provenance info. Client:LHCb Collaboration Achievement:Deployment of a Metadata Catalogue Service meeting the full demands of the LHCb experiment for metadata storage with significant UK input. High level Risks: Incompatibilities with future common technologies. Radical changes in metadata requirements due to fundamental change in philosophy of distributed production system and data model. Continued existence of ARDA project for those deliverables that envisage common solutions/middleware for all LHC experiments.

15. 15 LHCb Data Management: Level-1 Deliverables Deliverable LD1 (2004, end Q2?) Evaluation of the most efficient distinction between metadata and job provenance within the LHCb context as a contribution to ARDA. (Metric: public document) Deliverable LD2 (2004, end Q3?) Implementation of the prototype ARDA interface based on the existing LHCb book-keeping database as a backend. (Metric: Software release compatible with DIRAC and GANGA) (Dependencies: GANGA and DIRAC interfacing) Deliverable LD3 (2005, end Q1?) Tests and evaluation of prototype interface for 2005 Data Challenge. (Metric: interface deployed for DC05, user survey, document) (Dependencies: schedule for 2005 Data Challenge) Deliverable LD4 (2005, end Q3) Evaluate performance of alternative prototype implementations of interface to catalogues from other experiments. (Metric: document) Deliverable LD5 (2006, end Q1) Report on common hardware architectures for final catalogue. (Metric: report)

16. 16 LHCb Data Management: Level-1 Deliverables continued … Deliverable LD6 (2006, end Q3) Release prototype of catalogue. Grid enabled through an ARDA approved web-service interface. (Metric: software release, document) (Dependencies: ARDA agreed web-service interface) Deliverable LD7 (2007, end Q1) Prototype of general suite of tools to ensure a production quality service. These tools will monitor performance, availability and integrity of the catalogue. Identify solution for private metadata. (Metric: software release, document) Deliverable LD8 (2007, end Q2) Production release of catalogue and tool suite to support processing of real data. (Metric: Software release) Deliverable LD9 (2007, end Q3) Revised production (physics ready) release incorporating user feedback, updated documentation and private metadata functionality. (Metric: production software release) (Dependencies: availability of real data collisions at LHC)

17. ATLAS-specific plans RWL Jones for GridPP11, 14 th Sept 2004

18. ATLAS-specific plans All ATLAS plans are integrated with the GANGA developments and with the ATLAS eScience project and responsibilities Focus is on the development of a working fully- functional analysis environment Metadata is a key issue, identified in the proposal. This effort is now a major strand of the metadata project, and so will report in that strand but also to ATLAS Non-metadata GridPP manpower is Alvin Tan + 50% of Frederic Brochu

19. Release validation and integration GridPP (F Brochu++) has provided an essential component of the ATLAS validation and integration It is essential that this continue with LCG/EGEE and other releases (Objective 1.1) –Timing becomes less certain as these projects do not have a well-developed timetable of major releases to 2008 and have a bad track- record of functional releases!

20. New ATLAS Production System LCGNGGrid3LSF LCG exe LCG exe NG exe G3 exe LSF exe super ProdDB Data Man. System RLS jabber soap jabber Don Quijote Windmill Lexor AMI Capone Dulcinea Much of the problem is data management This must cope with >= 3 Grid catalogues The demands will be greater for analysis

21. Production Tools and Framework Alvin and Frederic have made important contribution to the ATLAS Grid production system. (Objective 1.2) –The overall system has the right structure, but the performance in DC2 is inadaquate –Major revision in the next 9 months –Periodic review thereafter The system must continually adapt to new middleware releases (executors for each system)

22. ATLANTIS – visualisation on the Grid Several applications need to be Grid- integrated The most important is ATLANTIS event and detector graphical display (Objective 2) –UK responsibility –Evolving workflows –Will help other application integration Work co-ordinated with effort in the ATLAS eScience programme

23. Schedule of Deliverables Deliverable A1 (2005 end Q3, Objective 1) –Refactored ATLAS production system integrated with EGEE middleware. –(Metric: software release) Deliverable A2 (2005 end Q4, Objective 2) –ATLANTIS Grid usage using plug-in operating the harvesting model for DC3 data and the metadata tools. ) – (Metric: demonstration, software release) Deliverable A3 (2006 end Q4, Objective 1) –Distributed analysis and production in DC4/Commissioning run. –(Metric: Analysis job submission, 90% or greater success rate.) Deliverable A4 (2006 end Q4, Objectives 1 and 2) –Releases for commissioning run. –(Metric: software release, documentation) Deliverable A5 (2007 end Q3, Objective 2) –Physics-ready release of ATLANTIS plug-in. –(Metric: software release) Deliverable A6 (2007 end Q3, Objectives 1 and 2) –Final release of Grid analysis and production tools for data-taking. –(Metric: software release)

24. 24