1. Outline: ARDA services LHCb mini-workshop on Data Management and Production Tools Ph.Charpentier m The ARDA RTAG m The ARDA services m The proposed project m Mapping to LHCb services

2. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management2 ARDA Mandate m

3. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management3 ARDA Schedule and Makeup m Alice: Fons Rademakers and Predrag Buncic Atlas: Roger Jones and Rob Gardner CMS: Lothar Bauerdick and Lucia Silvestris LHCb: Philippe Charpentier and Andrei Tsaregorodtsev LCG GTA: David Foster, stand-in Massimo Lamanna LCG AA: Torre Wenaus GAG: Federico Carminati

4. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management4 ARDA Distributed Analysis Services m Distributed Analysis in a Grid Services based architecture o ARDA Services should be OGSI compliant -- built upon OGSI middleware o Frameworks and applications use ARDA API with bindings to C++, Java, Python, PERL… P interface through UI/API factory -- authentication, persistent “session” o Fabric Interface to resources through CE, SE services P job description language, based on Condor ClassAds and matchmaking o Database(ses) through Dbase Proxy provide statefulness and persistence m We arrived at a decomposition into the following key services o Authentication, Authorization, Accounting and Auditing services o Workload Management and Data Management services o File and (event) Metadata Catalogues o Information service o Grid and Job Monitoring services o Storage Element and Computing Element services o Package Manager and Job Provenance services

5. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management5 ARDA Key Services for Distributed Analysis Numbers refer to time sequence of operations for a given use case: 1,2,3 Get access 4 Select dataset 5 Get PFMs 6 Submit job 7 Get files location 8 CE takes job 9,10 SW isinstalled 11 Progress checked 12,13 Get storage space 14,15 Store output

6. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management6 API to Grid services m Importance of API o Interface services to higher level software P Exp. framework P Analysis shells, e.g. ROOT P Grid portals and other forms of user interactions with environment P Advanced services e.g. virtual data, analysis logbooks etc o Provide experiment specific services P Data and Metadata management systems m Provide an API that others can project against o Benefits of common API to framework P Goes beyond “traditional” UIs à la GANGA, Grid portals, etc o Benefits in interfacing to analysis applications like ROOT et al o Process to get a common API b/w experiments --> prototype

7. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management7 On the road again m No “evolutionary” path from GT2-based grids o David Foster at June 24th POB P We have a complex software infrastructure that needs simplifying P ………………. P Cannot simply incrementally improve the software we have. d Based on Globus GT2 design (which is being replaced by OGSA GT3) m Augment LCG-1 and other grid services o ARDA Services deployed and run together with existing ones on LCG1 resources o Keep possibility to bridge to existing services if feasible P Grid connectivity rather than interoperability o Use invaluable experience of LCG1 deployment for deploying ARDA m ARDA provides decomposition into those services that address the LHC distributed analysis use cases m Recommendation: build early a prototype based on re-factoring existing implementations

8. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management8 ARDA Roadmap for Prototype m Prototype provides the initial blueprint o Do not aim for a full specification of all the interfaces m 4-prong approach: o Re-factoring of AliEn, Dirac and possibly other services into ARDA P Initial release with OGSI::Lite/GT3 proxy, consolidation of API, release P Implementation of agreed interfaces, testing, release o GT3 modeling and testing (in parallel) o Interfacing to LCG-AA software like POOL, analysis shells like ROOT P Also opportunity to “early” interfacing to complementary projects o Interfacing to experiments frameworks P metadata handlers, experiment specific services m Provide interaction points with community o Early releases and workshops every few months o Early strong feedback on API and services o Decouple from deployment issues

9. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management9 Experiments and LCG Involved in Prototyping m ARDA prototype would define the initial set of services and their interfaces. Timescale: spring 2004 m Important to involve experiments and LCG at the right level o Initial modeling of GT3-based services o Interface to major cross-exp packages: POOL, ROOT, PROOF, others o Program experiment frameworks against ARDA API, integrate with experiment environments o Expose services and API to other LHC projects to allow synergies o Spend appropriate effort to document, package, release, deploy m After the prototype is delivered, improve on o Scale up and re-engineer as needed: OGSI, databases, information services o Deployment and interfaces to site and grid operations, VO management etc o Build higher-level services and experiment specific functionality o Work on interactive analysis interfaces and new functionalities

10. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management10 Possible Strawman m Strawman workplan for ARDA prototype

11. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management11 Setting up the project m Propose ARDA to become now an LCG project o Project should start with a definition of the work areas identifying where the effort will come from m Core development team: 2-3 *good* (experienced) people plus 1 person from each experiment m Estimate roughly total effort of some 10-15 people for the 6- month timescale to be practical m Relevant experience and manpower coming from AliEn & Dirac developers, other LHC experiments, GTA, AA, … o Alice & LHCb needs to evaluate the impact on AliEn/Dirac planning and makes a strong commitment to provide the relevant expertise

12. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management12 Outcome of the SC2 presentation (October 3rd) m Good support from LCG management, even from EGEE reps m Surprise of CERN management (what, this is Grid developers’ job…) m Strange attitude of ATLAS (to a lesser extend CMS) o Their problem is that they have a lot of projects on this (Clarens, Dial, …) o Will take time to get their people onboard m Written report expected end October m In parallel, discussion are ongoing to set up the project m Should be officially launched beginning of November m IMPORTANT: there must be a strong participation besides ALICE… Otherwise, it will be plain AliEn o We should not hesitate to invest, it may pay back…

13. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management13 LHCb view of ARDA services Bookkeeping Database Ganga Production Manager Dirac SW Installation bbftp Dirac agent Castor Production Database LHCb Production Account LSF Handled by the running script

14. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management14 Dependencies of services for LHCb m Gaudi will use POOL for data storage o POOL has to use the LHCb file catalog (or vice-versa) o File catalog has to be decoupled from Bookkeeping o Can be XML, mySQL, RLS…. or ARDA-compatible catalog? o XML has to be used to test Gaudi/POOL (not suitable in production) m File catalog interfaced to o User interfaces (GANGA, shell commands) o Data Management service - to be defined (file replication) P Directly handled by Dirac agent in a first instance? m Bookkeeping o Is it possible to define an interface? o Experiment dependent interface? Is it a problem? o Could use the existing BKDB to start with. Evaluate others (ARDA prototype)

15. PhC, 16/10/03 ARDA services, LHCb workshop on Data Management15 Dependencies of services for LHCb m Software installation o Should software be yet another file in the catalog, i.e. use the data management service to install it (a la AliEn)? o Otherwise is network connectivity necessary? m Authentication, authorization o Should it be fully traceable? o Should CEs trust the workload management (a la Dirac?) o How to use a Grid security system and is it feasible at all? o This is one of the biggest issues (for analysis) as Computer Centres don’t want to give access to (even trusted) generic accounts o Could/should Dirac transfer credentials? How?