1. The Ganga User Interface for Physics Analysis on Distributed Resources
Alexander Soroko (Oxford University)
Motivation and Background
Components
Submission of Analysis Jobs
Plans and Conclusions

2. Ganga-related information regularly updated on web site
The Project
Ganga is being developed as a joint project between ATLAS and LHCb
Began in the UK supported by GridPP, important collaborations with US colleagues
Current contributors are:
Code Developers: A.Soroko (Oxford), K.Harrison (Cambridge), C.L.Tan (Birmingham), J.Moscicki (CERN), J.Martyniak (Imperial), A.Maier (CERN)
Technical direction: D.Adams (BNL), U.Egede (Imperial), P.Charpentier (CERN), P.Mato (CERN)
GridPP coordination: G.N.Patrick (RAL), R.W.L.Jones (Lancaster)
Ganga-related information regularly updated on web site
CHEP 2004
18/09/2018

3. Motivation and Background
ATLAS and LHCb develop applications within a common framework: Gaudi/Athena
Both collaborations aim to exploit potential of Grid for large-scale, data-intensive distributed computing
Simplify management of analysis and production jobs for end-user physicists by developing tools for accessing Grid and other services
Built-in knowledge of how Gaudi/Athena works: Gaudi/Athena and Grid Alliance (Ganga)
CHEP 2004
18/09/2018

4. Motivation and Background
Athena/Gaudi
Ganga
GUI/CLI
JobOptions
Algorithms
Collective
&
Resource
Grid
Services
Histograms
Monitoring
Results
CHEP 2004
18/09/2018

5. General Outlook
CHEP 2004
18/09/2018

6. Application Interface UI Abstract Definition
Software Bus Design
External
Components
Core
Specialised
Job Registry
Data Interface
Software Bus
CLI
GUI
PyRoot
Gaudi Python
PyCMT
PyAMI
LHCb Bookkeeping DB
Job Splitters
Job Options Editor
Application Interface
Job Interface
UI Abstract Definition
File Handlers
Application Handlers
Job Handlers
AJDL
Specialised and External components work like plug-ins and not necessary present in the installation
For more info about Software Bus see poster of Wim LAVRIJSEN (session 24, track t3)
CHEP 2004
18/09/2018

7. UI Abstract Definition Application Interface
Core Components
CLI
GUI
Specialised
Components
Software Bus
Job Splitters
Job Options Editor
File Handlers
Application Handlers
Job Handlers
Core
Components
UI Abstract Definition
Job Interface
Application Interface
Data Interface
Job Registry
External
Components
LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyAMI
PyCMT
AJDL
CHEP 2004
18/09/2018

8. Core Components
Core Components define classes to support basic functionality of Ganga
They provide generic Interfaces for the:
Job (what is submitted)
Application (the executable run by the Job)
Data (input and output data to the application)
Job Registry component provides the means to store and recover job information
UI Abstract Definition component enables dynamic GUI construction
Core components are not bound to any specific type of application (e.g. Gaudi/Athena)
Core components are self consistent and can be used outside of Ganga to build other applications
CHEP 2004
18/09/2018

9. Specialised Components
CLI
GUI
Specialised
Components
Core
Components
Job Handlers
UI Abstract Definition
Application Handlers
Job Interface
Software Bus
File Handlers
Application Interface
Job Options Editor
Data Interface
Job Splitters
Job Registry
External
Components
LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyAMI
PyCMT
AJDL
CHEP 2004
18/09/2018

10. Specialised Components
Specialised components like Job, Application and File Handlers implement interfaces provided by Core components for job, application and data correspondingly
Job-handling components
Manage job submission to different batch systems (EDG/LCG, LSF, PBS, DIRAC WMS) and local PCs
Perform job configuration e.g., create JDL files
Generate work flow scripts to be executed on the target batch system
Application-handling components
Provide configuration templates covering specific tasks e.g., setting up environment, discovery of user dlls etc
File-handling components
Transfer input & output files between worker nodes and different storage elements
CHEP 2004
18/09/2018

11. Job Options Editor (JOE)
Difficulties in creating and manipulating application configuration files within the Gaudi/Athena frameworks (typically known as job option files):
Knowledge of available options/values
Knowledge of job options syntax
Spelling errors
JOE makes these difficulties easy with convenient GUI
Collapsible tree representation allows options, attributes or included files of interest to be viewed on demand.
Multiple job option files may be open for simultaneously editing using window tabs.
CHEP 2004
18/09/2018

12. View of job options file
JOE GUI
View of job options file
Main window of JOE
CHEP 2004
18/09/2018

13. Job Splitters Job Handling module Repository Subjob 1
Template Job
Repository
Splitting script 1
Splitting script 2
Splitting script …
Job Handling module
Subjob 1
Subjob 2
Subjob 3
Subjob 4
Subjob …
Selects or creates
CHEP 2004
18/09/2018

14. Application Interface UI Abstract Definition
External Components
Core
Components
Specialised
Job Registry
Data Interface
Software Bus
CLI
GUI
Job Splitters
Job Options Editor
Application Interface
Job Interface
UI Abstract Definition
File Handlers
Application Handlers
Job Handlers
External
Components
LHCb Bookkeeping DB
PyRoot
Gaudi Python
PyAMI
PyCMT
AJDL
CHEP 2004
18/09/2018

15. List of available datasets
LHCb Bookkeeping DB
List of available datasets
Selection criterion
CHEP 2004
18/09/2018

16. ATLAS: AJDL Components
AJDL is Abstract Job Description Language, used by DIAL (ATLAS analysis framework)
The building blocks of an AJDL job are:
Application (described by its name and version, taken from central repository)
Task (collection of files required by the application)
Dataset (the user issues an SQL query to a dataset catalog)
Preferences (provide the means to fine-tune the AJDL job)
CHEP 2004
18/09/2018

17. Submission of LHCb Analysis Job
Prepare/compile/select packages
Mainly outside of Ganga
Edit algorithm workflow and parameters of algoritms
JOE
Select input datasets
Bookkeeping DB interface
Split job
Job splitters
Press “Submit” button
Watch job status
See job output in the job folder
CHEP 2004
18/09/2018

18. How Ganga Works Ganga Submit job Grid/Batch System Gatekeeper
Send job output
Grid/Batch System Gatekeeper
DIRAC WMS
DIAL
Worker Nodes
Files Transfer
Job script
JDL file
Job Options file
Storage Element
Get Monitoring Info
Get job output
Send
Ganga
Job object
Job object
Job object
Job object
Job object
Job Factory (Job Registry Class)
Job Options Editor
Data Selection
Strategy Selection (e.g., Splitting Scripts)
Job Requirements (Resources)
Job Options DB
Bookkeeping DB
CHEP 2004
18/09/2018

19. Job was submitted to DIRAC WMS
GUI: Job Submission
Job was submitted to DIRAC WMS
For more information about DIRAC see talk of Andrei TSAREGORODTSEV (session 7 track t4)
CHEP 2004
18/09/2018

20. Plans and Conclusions
Ganga has been developed as a joint project between ATLAS and LHCb
Ganga modular design:
Provides uniform user interface to different distributed resources
Enables experiment customisation
Plans:
Convenient CLI (scripting)
Writing interfaces to other analysis services (gLite)
Building remote Ganga services running under control of a local client
GUI interface for the above client
Services to provide centralised persistency of user jobs
Ganga session persistency (history, GUI preferences)
Roaming access to user profiles
The intention is to make life easier for users
CHEP 2004
18/09/2018