1. EGEE is a project funded by the European Union under contract IST-2003-508833 “ARDA status” Dietrich Liko / CERN XXVII HTASC, 10 September 2004 www.eu-egee.org http://cern.ch/arda cern.ch/lcg

2. XXVII HTASC: ARDA status Dietrich Liko 2 Overview The EGEE project ARDA in a nutshell  Experiments  Middleware Highlights from the 4 experiment prototypes  CMS, ATLAS, LHCb and ALICE ARDA-related workshops Conclusion

3. XXVII HTASC: ARDA status Dietrich Liko 3 The EGEE project Create a European-wide Grid production quality infrastructure for multiple sciences Profit from current and planned national and regional Grid programmes, building on  the results of existing projects such as DataGrid (EDG), LCG and others  EU Research Network and industrial Grid developers Support Grid computing needs common to the different communities  integrate the computing infrastructures and agree on common access policies Exploit International connections (US and AP)  Provide interoperability with other major Grid initiatives such as the US NSF Cyberinfrastructure, establishing a worldwide Grid infrastructure Leverage national resources in a more effective way 70 leading institutions in 27 countries (including Russia and US)

4. XXVII HTASC: ARDA status Dietrich Liko 4 Strong links already between EDG and LCG. It will continue in the scope of EGEE The core infrastructure of the LCG and EGEE grids will be operated as a single service  LCG has many US and Asia partners  EGEE includes other sciences  Substantial part of infrastructure common to both Parallel production lines  LCG-2: 2004 data challenges  EGEE Prototype of new MW ARDA playground for the LHC experiments EGEE WS MW EGEE-2EGEE-1LCG-2LCG-1 VDT/EDG ARDA EGEE and LCG

5. XXVII HTASC: ARDA status Dietrich Liko 5 Starting point for ARDA New service decomposition  Strong influence by the Grid system developed by the ALICE experiment, Alien and used by a wide scientific community (not only HEP) Role of new technology, experiences of the past …  Web service framework Interfacing of middleware for use in the experiment frameworks  Systems are already in use today Early deployment of (a series of) prototypes  functionality and coherence EGEE Middleware ARDA project

6. XXVII HTASC: ARDA status Dietrich Liko 6 ARDA in a nutshell ARDA is an LCG project  main activity is to enable LHC analysis on the grid ARDA is contributing to EGEE NA4  uses the entire CERN NA4-HEP resource Work is based on last years experience/components  Grid projects (LCG, VDT, EDG …)  Experiments middleware/tools (Alien, Dirac, GAE, Octopus, Ganga, Dial,…) Interface with the new EGEE middleware (gLite)  Use the grid software as it matures  Key player in the evolution from LCG2 to the EGEE infrastructure  Verify the components in an analysis environments  Provide early and continuous feedback

7. XXVII HTASC: ARDA status Dietrich Liko 7 ARDA and HEP experiments Interface with the HEP Experiments  Every experiment has different implementations of the standard services  Help in adapting/interfacing (direct help within the experiments) Move from current production environments …  Few expert users  Coordinated update and read actions  Used mainly in so-called data challanges …to an analysis environment  Many users (Robustness might be an issue)  Concurrent “read” actions (Performance will be more and more an issue)  Used by all physicists for their analysis

8. XXVII HTASC: ARDA status Dietrich Liko 8 Working model Development of one prototype per experiment  ARDA emphasis is to enable each of the experiment to do its job  A Common Application Layer might emerge in future Provide a forum for discussion  Comparison on results/experience/ideas  Interaction with other projects  … Organizes workshops for interaction with community

9. XXVII HTASC: ARDA status Dietrich Liko 9 ARDA team Massimo Lamanna Birger Koblitz Derek Feichtinger Andreas Peters Dietrich Liko Frederik Orellana Julia Andreeva Juha Herrala Andrew Maier Kuba Moscicki Andrey Demichev Viktor Pose Wei-Long Ueng Tao-Sheng Chen LHCb CMS ATLAS ALICE Russia Taiwan Experiment interfaces Piergiorgio Cerello (ALICE) David Adams (ATLAS) Lucia Silvestris (CMS) Ulrik Egede (LHCb)

10. XXVII HTASC: ARDA status Dietrich Liko 10 Milestones End-To-End Prototype activity MilestoneDateDescription 1.6.18Dec 2004E2E prototype for each experiments (4 prototypes), capable of analysis (or advanced production) 1.6.19Dec 2005E2E prototype for each experiments (4 prototypes), capable of analysis and production

11. XXVII HTASC: ARDA status Dietrich Liko 11 Middleware architecture Many components User access by GAS

12. XXVII HTASC: ARDA status Dietrich Liko 12 Middleware Prototype Source: http://egee-jra1.web.cern.ch/egee-jra1/Prototype/testbed.htmhttp://egee-jra1.web.cern.ch/egee-jra1/Prototype/testbed.htm CE (lxn5210) VOMS/myProxy (lxn5210) WN (lxn5211) SE dCache (lxn5208) SE Castor (lxn5209) Core services (lxn5219) Database, proxy, ldap (lxn5220) GAS (lxn5216)

13. XXVII HTASC: ARDA status Dietrich Liko 13 Middleware Prototype Available for us since May 18 th  In the first month, many problems connected with the stability of the service and procedures  At that point just a few worker nodes available  A second site (Madison) available since end of June  CASTOR access to the actual data store No. of CPUs will increase  50 as a target for CERN, hardware available Nr. of sites will increase

14. XXVII HTASC: ARDA status Dietrich Liko 14 LHCb Main component: GANGA  GUI access to the Grid  Enable physicists to analyze the data being produced during 2004 for their studies  It naturally matches the ARDA mandate  Deployment of the prototype where the LHCb data will be is essential (CERN, RAL, …) At the start the emphasis is to validate the tool  Focus on overall usability  Splitting and merging functionality for users jobs DIRAC (LHCb production grid)  Convergence with GANGA / components / experience  Submit jobs to DIRAC using GANGA

15. XXVII HTASC: ARDA status Dietrich Liko 15 GANGA Gaudi/Athena aNd Grid Alliance Gaudi/Athena: LHCb/ATLAS frameworks Single “desktop” for a variety of tasks Help configuring and submitting analysis jobs Keep track of what they have done, hiding completely all technicalities GAUDI Program GANGA GUI JobOptions Algorithms Collective & Resource Grid Services Histograms Monitoring Results Grid Services GANGA UI BkSvc Bookkeeping Service WorkLoad Manager SE File catalog WLMProSvcMonitor Internal Model Profile Service GAUDI Program Instr. CE

16. XXVII HTASC: ARDA status Dietrich Liko 16 LHCb Use of the gLite testbed  Simple DaVinci jobs from GANGA to gLite  “Regular” DaVinci jobs onto gLite Other contributions  GANGA interface to Condor (Job submission) and Condor DAGMAN for splitting/merging and error recovery  GANGA Release management and software process CVS, Savannah,…  Contributions to DIRAC  Metadata catalogue tests Performance tests Collaborators in Taiwan (ARDA + local DB know-how on Oracle)

17. XXVII HTASC: ARDA status Dietrich Liko 17 CMS The CMS system within ARDA is still under discussion  Milestone 1.6.4 late by 3 months Key issue (Data management)  Provide easy access (and possibly sharing) of data for the CMS users Exploratory/preparatory activity  Successful ORCA job submission to gLite. Now investigating with the package manager  Access to files directly from CASTOR  gLite file catalog

18. XXVII HTASC: ARDA status Dietrich Liko 18 CMS Data Management RefDB  RefDB is the bookkeeping engine to plan and steer the production across different phases simulation, reconstruction to some degree into the analysis phase). This service is under test  It contained all information except file physical location (RLS) info related to the transfer management system (TMDB)  The actual mechanism to provide these data to analysis users is under discussion  Measuring performances underway (similar philosophy as for the LHCb Metadata catalog measurements) Updates RefDB McRunjob T0 worker nodes GDB castor pool Tapes Export Buffers Transfer agent RLSTMDB Reconstruction instructions Reconstruction jobs Reconstructed data Reconstructed data Checks what has arrived Updates Summaries of successful jobs RefDB in CMS DC04

19. XXVII HTASC: ARDA status Dietrich Liko 19 ATLAS The ATLAS system within ARDA has been agreed  ATLAS has a complex strategy for distributed analysis, addressing different area with specific projects (www.usatlas.bnl.gov/ADA)  Starting point is: DIAL analysis model (high level web services) DIAL on gLite OK (Evolution of the DIAL demo) ATHENA to gLite OK First skeleton of high level services

20. XXVII HTASC: ARDA status Dietrich Liko 20 DIAL - Distributed Interactive Analysis of Large datasets

21. XXVII HTASC: ARDA status Dietrich Liko 21 ATLAS AMI Tests The AMI metadata catalog is a key component  Robustness and performance tests from ARDA  Very good relationship with the ATLAS Grenoble group  Discussions on technology (EGEE JRA1 in the loop) In the start up phase, ARDA provided help in developing ATLAS tools (ATCOM and CTB)

22. XXVII HTASC: ARDA status Dietrich Liko 22 ALICE USER SESSION PROOF SLAVES PROOF PROOF MASTER SERVER PROOF SLAVES Site A Site C Site B PROOF Analysis system Based on ROOT The ALICE/ARDA will evolve the ALICE analysis system (SuperComputing 2003)

23. XXVII HTASC: ARDA status Dietrich Liko 23 Forward Proxy Rootd Proofd Grid/Root Authentication Grid Access Control Service TGrid UI/Queue UI Proofd StartupPROOFClient PROOFMaster Slave Registration/ Booking- DB Site PROOF SLAVE SERVERS Site A PROOF SLAVE SERVERS Site BPROOFSteer Master Setup New Elements Grid Service Interfaces Grid File/Metadata Catalogue Client retrieves list of logical file (LFN + MSN) Booking Request with logical file names “Standard” Proof Session Slave ports mirrored on Master host Optional Site Gateway Master Client Grid-Middleware independend PROOF Setup Only outgoing connectivity Status report an as a demo during the workshop by A. Peters

24. XXVII HTASC: ARDA status Dietrich Liko 24 ALICE Where to improve:  Strong requests on networking (inbound connectivity)  Heavily connected with the middleware services  “Inflexible” configuration  No chance to use PROOF on federated grids like LCG in AliEn  User libraries distribution Activity on PROOF  Robustness and Error recovery Grid activity:  C++ access library on gLite  IO library contributions

25. XXVII HTASC: ARDA status Dietrich Liko 25 ARDA workshops  1 st ARDA workshop (January 2004 at CERN; open)  2 nd ARDA workshop (June 21-23 at CERN; by invitation) “The first 30 days of EGEE middleware” Main focus on LHC experiments and EGEE JRA1 (Glite)  NA4 meeting mid July NA4/JRA1 and NA4/SA1 sessions organised by M. Lamanna and F. Harris EGEE/LCG operations new ingredient!  3 rd ARDA workshop (October 2004; open) “The LCG ARDA prototypes”  EGEE Conference meeting mid November NA4/JRA1 and NA4/SA1 sessions organised by M. Lamanna and F. Harris

26. XXVII HTASC: ARDA status Dietrich Liko 26 “The first 30 days of the EGEE middleware” ARDA workshop New situation:  gLite middleware becoming available  LCG ARDA project started  Experience + need of technical discussions New format:  “Small” (30 participants vs 150 in January), by invitation only…  ARDA team + experiments interfaces  EGEE Glite team (selected persons)  Experiments technical key persons  Technology experts  NA4/EGEE links (4 persons)  EGEE PTF chair Info on the web:  URL:http://lcg.web.cern.ch/LCG/peb/arda/LCG_ARDA_Workshops.htm

27. XXVII HTASC: ARDA status Dietrich Liko 27 Workshop executive summary By invitation  positive technical discussions  not everybody could be invited Emphasis on experiments  demonstrate their status and their plans MW architecture document available   missing a detailed description of gLite Important messages from ARDA  Resources: CPUs and sites  Procedure: Registration as an example  Stability: Service crashes Next workshop will be open  October 20-21

28. XXVII HTASC: ARDA status Dietrich Liko 28 Important messages from the workshop Prototype approach OK (iterate!)  Priority on new functionality  Prepare larger infrastructure  Expose the API of all services GAS useful - Grid Access based on Web Services  Direct access to components is also important DB access via Web Services - unclear File Catalogue - Read-only files Metadata catalogues - Many projects already active, convergence unclear Data Management tools - can TMdb be implemented with gLite? Package management - interesting but unclear priority

29. XXVII HTASC: ARDA status Dietrich Liko 29 Conclusions ARDA is up and running  Since April 1 st preparing the ground for the experiments prototypes  Definition of the detailed work program  Contributions in the experiment-specific domain  Prototype activity started Next important steps  (More) real users  Need of more hardware resources  Both important for December 2004 milestone Stay tuned (and attend the workshop in October )