ATLAS/LHCb GANGA DEVELOPMENT Introduction Requirements Architecture and design Interfacing to the Grid Ganga prototyping A. Soroko (Oxford), K. Harrison (Cambridge)

16/09/2002GridPP meeting, IC 2 Introduction ATLAS and LHCb develop applications within a common framework: Gaudi/Athena Both collaborations aim to exploit potential of Grid for large-scale, data-intensive distributed computing ATLAS and LHCb develop applications within a common framework: Gaudi/Athena Both collaborations aim to exploit potential of Grid for large-scale, data-intensive distributed computing Simplify management of analysis and production jobs for end-user physicists by developing tool for accessing Grid services with built-in knowledge of how Gaudi/Athena works: Gaudi/Athena and Grid Alliance (GANGA) Simplify management of analysis and production jobs for end-user physicists by developing tool for accessing Grid services with built-in knowledge of how Gaudi/Athena works: Gaudi/Athena and Grid Alliance (GANGA)

16/09/2002GridPP meeting, IC 3 Introduction GAUDI Program GANGA GUI JobOptions Algorithms Collective & Resource Grid Services Histograms Monitoring Results GANGA: Gaudi/Athena and Grid Alliance First ideas for GANGA were presented by P.Mato and C.Tull in summer 2001 An Atlas/LHCb design team is being set up, including US representatives

16/09/2002GridPP meeting, IC 4 General requirements for the GUI The user will interact with a single application integrating all stages of job life-time. He will be able to restore his workspace (list of files, tools state, jobs in preparation) at the beginning of each session. The GUI will be similar to work with for both the Grid and a local network The user will be able to access the interface not only from the computer with the Grid UI program running, but also from a remote thin client.

16/09/2002GridPP meeting, IC 5 Required functionality Job preparation and configuration Resource booking Job submission User can choose between Grid and local resource management system Job monitoring and control Can we use GRM and PROVE and/or modified Gaudi services? GUI for the resource browsing VO active services CEs SEs Query existing files in the Grid GUI for data management tools e.g., Dataset registration to the Grid (used by Production Manager) Copy file from a Computing Element to a Storage Element Replication of files

16/09/2002GridPP meeting, IC 6 Job preparation and configuration : Determine job requirements in terms of software products needed: executables, libraries, databases, etc. Get access to the Job Configurations DB Common configurations could be stored in a database and retrieved using high-level commands User would have possibility of modifying settings and storing personalised configurations in his/her own area Perform job configuration: select algorithms to run and set properties specify input event data requested output, etc Provide graphical tools for editing default Job Options files Contact the Gaudi Bookkeeping Database and the Grid Replica Catalogue to obtain the list of Logical File Names (LFNs) from high- level physics selection criteria Automated generation of JDL scripts for job submission Required functionality

16/09/2002GridPP meeting, IC 7 General Requirements for the Architecture Simplicity of implementation Portability (platform independence) Rich functionality Modularity, which allows for Extensibility Should provide interactivity Underlying GRID services (GLOBUS toolkit) GRID middleware (EDG, PPDG,…) Application specific layer (Athena/Gaudi, …) GUI interface OS and Network services Multilayered Grid architecture GANGA

16/09/2002GridPP meeting, IC 8 Design of GANGA Two ways of implementation have been discussed at Coseners House meeting: Based on one of the general-purpose grid portals (not tied to a single application/framework): Alice Environment (AliEn) Grid Enabled Web eNvironment for Site-Independent User Job Submission (GENIUS) Grid access portal for physics applications (Grappa) Simulation for LHCb and its Integrated Control Environment (SLICE) Based on the concept of Python bus (P. Mato): use different modules whichever are required to provide full functionality of the interface use Python to glue this modules, i.e., allow interaction and communication between them Different existing implementations have been studied and outcome is been summarized in the Technical Survey (K. Harrison and A. Soroko) We have decided that a new development using Python software bus is better suited to the aims of ATLAS and LHCb

16/09/2002GridPP meeting, IC 9 Python Bus Design Server Bookkeeping DB Production DB EDG UI PYTHON SW BUS XML RPC server XML RPC module GANGA Module OS Module Athena\ GAUDI GaudiPython PythonROOT PYTHON SW BUS GUI Job Configuration DB Remote user (client) Local Job DB LAN/WAN GRID LRMS

16/09/2002GridPP meeting, IC 10 Interfacing to the Grid Job preparation and configuration Contact RC to get list of LFNs relevant to that job Contact RB to get list of suitable CE Resource booking Contact RB??? Job submission Contact GSI and/or Myproxy server to get proxy certificate Contact RB to submit a job Job monitoring and control Contact LB service to get logging information Contact RB to get job status Resource browsing Contact Globus MDS (meta-computing directory service) using LDAP search commands Data management Contact RB to get job output Use GridFTP and globus-url-copy for file transfers Contact RM to register and replicate files GANGA functionality requires use of Grid tools in many cases:

16/09/2002GridPP meeting, IC 11 Interfacing to the Grid Job class GANGA kernel module Jobs registry class XML RPC Data management service Job submission service Job monitoring service Security service dg-job-list-match dg-job-submit dg-job-cancel dg-job-list-match dg-job-submit dg-job-cancel grid-proxy-init MyProxy ? GSI ? grid-proxy-init MyProxy ? GSI ? dg-job-status dg-job-get-logging-info GRM/PROVE dg-job-status dg-job-get-logging-info GRM/PROVE edg-replica-manager dg-job-get-output globus-url-copy GDMP? edg-replica-manager dg-job-get-output globus-url-copy GDMP? EDG UI

16/09/2002GridPP meeting, IC 12 Ganga Prototyping Embedded Python interpreter Tree of user jobs Job options for selected job

16/09/2002GridPP meeting, IC 13 Ganga Prototyping GUI is created using wxPython extension module Access to the Gaudi Job Configuration DB is implemented with the xmlrpclib module User can browse and create Job Options files using this DB Serialization of objects (user jobs) is implemented with the Python pickle module Python interpreter is embedded into the GUI and allows user to configure interface from the command line GRID stuff is under development at the moment and is oriented on EDG testbed 1.2

16/09/2002GridPP meeting, IC 14 Conclusions We intend to have our prototype interfaced to both production and analysis Gaudi-aware jobs. Many details of GANGA architecture are in the experimental stage and require further approval from the Atlas/LHCb design team (e.g. XML RPC layer between GANGA client and EDG UI). Most likely many of GANGA modules will not be presented in the prototype. For example, possible design of a module responsible for the HTML-based access to the interface has not been developed yet. Some questions remain related to the security issues, e.g., could we have GSI compiled for different platforms ? In order to implement interfacing to the Grid we need Python API to the EDG UI. Should we create this API ourselves?