The Ganga User Interface for Physics Analysis on Distributed Resources Alexander Soroko (Oxford University) Motivation and Background Components Submission of Analysis Jobs Plans and Conclusions

Ganga-related information regularly updated on web site The Project Ganga is being developed as a joint project between ATLAS and LHCb Began in the UK supported by GridPP, important collaborations with US colleagues Current contributors are: Code Developers: A.Soroko (Oxford), K.Harrison (Cambridge), C.L.Tan (Birmingham), J.Moscicki (CERN), J.Martyniak (Imperial), A.Maier (CERN) Technical direction: D.Adams (BNL), U.Egede (Imperial), P.Charpentier (CERN), P.Mato (CERN) GridPP coordination: G.N.Patrick (RAL), R.W.L.Jones (Lancaster) Ganga-related information regularly updated on web site http://ganga.web.cern.ch/ganga CHEP 2004 18/09/2018

Motivation and Background ATLAS and LHCb develop applications within a common framework: Gaudi/Athena Both collaborations aim to exploit potential of Grid for large-scale, data-intensive distributed computing Simplify management of analysis and production jobs for end-user physicists by developing tools for accessing Grid and other services Built-in knowledge of how Gaudi/Athena works: Gaudi/Athena and Grid Alliance (Ganga) CHEP 2004 18/09/2018

Motivation and Background Athena/Gaudi Ganga GUI/CLI JobOptions Algorithms Collective & Resource Grid Services Histograms Monitoring Results CHEP 2004 18/09/2018

General Outlook CHEP 2004 18/09/2018

Application Interface UI Abstract Definition Software Bus Design External Components Core Specialised Job Registry Data Interface Software Bus CLI GUI PyRoot Gaudi Python PyCMT PyAMI LHCb Bookkeeping DB Job Splitters Job Options Editor Application Interface Job Interface UI Abstract Definition File Handlers Application Handlers Job Handlers AJDL Specialised and External components work like plug-ins and not necessary present in the installation For more info about Software Bus see poster of Wim LAVRIJSEN (session 24, track t3) CHEP 2004 18/09/2018

UI Abstract Definition Application Interface Core Components CLI GUI Specialised Components Software Bus Job Splitters Job Options Editor File Handlers Application Handlers Job Handlers Core Components UI Abstract Definition Job Interface Application Interface Data Interface Job Registry External Components LHCb Bookkeeping DB PyRoot Gaudi Python PyAMI PyCMT AJDL CHEP 2004 18/09/2018

Core Components Core Components define classes to support basic functionality of Ganga They provide generic Interfaces for the: Job (what is submitted) Application (the executable run by the Job) Data (input and output data to the application) Job Registry component provides the means to store and recover job information UI Abstract Definition component enables dynamic GUI construction Core components are not bound to any specific type of application (e.g. Gaudi/Athena) Core components are self consistent and can be used outside of Ganga to build other applications CHEP 2004 18/09/2018

Specialised Components CLI GUI Specialised Components Core Components Job Handlers UI Abstract Definition Application Handlers Job Interface Software Bus File Handlers Application Interface Job Options Editor Data Interface Job Splitters Job Registry External Components LHCb Bookkeeping DB PyRoot Gaudi Python PyAMI PyCMT AJDL CHEP 2004 18/09/2018

Specialised Components Specialised components like Job, Application and File Handlers implement interfaces provided by Core components for job, application and data correspondingly Job-handling components Manage job submission to different batch systems (EDG/LCG, LSF, PBS, DIRAC WMS) and local PCs Perform job configuration e.g., create JDL files Generate work flow scripts to be executed on the target batch system Application-handling components Provide configuration templates covering specific tasks e.g., setting up environment, discovery of user dlls etc File-handling components Transfer input & output files between worker nodes and different storage elements CHEP 2004 18/09/2018

Job Options Editor (JOE) Difficulties in creating and manipulating application configuration files within the Gaudi/Athena frameworks (typically known as job option files): Knowledge of available options/values Knowledge of job options syntax Spelling errors JOE makes these difficulties easy with convenient GUI Collapsible tree representation allows options, attributes or included files of interest to be viewed on demand. Multiple job option files may be open for simultaneously editing using window tabs. CHEP 2004 18/09/2018

View of job options file JOE GUI View of job options file Main window of JOE CHEP 2004 18/09/2018

Job Splitters Job Handling module Repository Subjob 1 Template Job Repository Splitting script 1 Splitting script 2 Splitting script … Job Handling module Subjob 1 Subjob 2 Subjob 3 Subjob 4 Subjob … Selects or creates CHEP 2004 18/09/2018

Application Interface UI Abstract Definition External Components Core Components Specialised Job Registry Data Interface Software Bus CLI GUI Job Splitters Job Options Editor Application Interface Job Interface UI Abstract Definition File Handlers Application Handlers Job Handlers External Components LHCb Bookkeeping DB PyRoot Gaudi Python PyAMI PyCMT AJDL CHEP 2004 18/09/2018

List of available datasets LHCb Bookkeeping DB List of available datasets Selection criterion CHEP 2004 18/09/2018

ATLAS: AJDL Components AJDL is Abstract Job Description Language, used by DIAL (ATLAS analysis framework) The building blocks of an AJDL job are: Application (described by its name and version, taken from central repository) Task (collection of files required by the application) Dataset (the user issues an SQL query to a dataset catalog) Preferences (provide the means to fine-tune the AJDL job) CHEP 2004 18/09/2018

Submission of LHCb Analysis Job Prepare/compile/select packages Mainly outside of Ganga Edit algorithm workflow and parameters of algoritms JOE Select input datasets Bookkeeping DB interface Split job Job splitters Press “Submit” button Watch job status See job output in the job folder CHEP 2004 18/09/2018

How Ganga Works Ganga Submit job Grid/Batch System Gatekeeper Send job output Grid/Batch System Gatekeeper DIRAC WMS DIAL Worker Nodes Files Transfer Job script JDL file Job Options file Storage Element Get Monitoring Info Get job output Send Ganga Job object Job object Job object Job object Job object Job Factory (Job Registry Class) Job Options Editor Data Selection Strategy Selection (e.g., Splitting Scripts) Job Requirements (Resources) Job Options DB Bookkeeping DB CHEP 2004 18/09/2018

Job was submitted to DIRAC WMS GUI: Job Submission Job was submitted to DIRAC WMS For more information about DIRAC see talk of Andrei TSAREGORODTSEV (session 7 track t4) CHEP 2004 18/09/2018

Plans and Conclusions Ganga has been developed as a joint project between ATLAS and LHCb Ganga modular design: Provides uniform user interface to different distributed resources Enables experiment customisation Plans: Convenient CLI (scripting) Writing interfaces to other analysis services (gLite) Building remote Ganga services running under control of a local client GUI interface for the above client Services to provide centralised persistency of user jobs Ganga session persistency (history, GUI preferences) Roaming access to user profiles The intention is to make life easier for users CHEP 2004 18/09/2018