ATLAS-specific functionality in Ganga - Requirements for distributed analysis - ATLAS considerations - DIAL submission from Ganga - Graphical interfaces - Next steps K. Harrison CERN, 26th May 2005

26th May Requirements for distributed analysis -General requirements for distributed analysis similar for ATLAS and LHCb  Deploy and run software based on Gaudi/Athena framework, and developed in terms of CMT packages  Locate data for analysis  Run on a variety of backends  Manage job output -Differences in the choices of how to do the above  Need for experiment-specific functionality in Ganga

26th May ATLAS considerations (1) -Installation of ATLAS software is performed using pacman  Binary installation is 8-10Gb, so installation not to be taken lightly  Pacman tool also being tested by LHCb -Data-management system (Don Quijote) for ATLAS still under development  Must manage data distributed over different Grids Today: LCG, NorduGrid, Grid 3 (US)  Have metadata catalogue (AMI) similar to Bookkeeping Database of LHCb, and various other catalogues

26th May ATLAS considerations (2) -Two systems for distributed data processing developed  Production system used for collaboration-wide managed activities Similarities with DIRAC in LHCb  ATLAS Distributed Analysis (ADA) system, based on DIAL  Overlaps between the two systems, and optimal way of using them still being understood -In Ganga, two systems can be treated as different backends

26th May The production database, which contains abstract job definitions; -The windmill supervisor that reads the production database for job definitions and present them to the different GRID executors in an easy-to-parse XML format; -The Executors, one for each GRID flavor, that receive the job-definitions in XML format and convert them to the job description language of that particular GRID; -Don Quijote, the Atlas Data Management System, moves files from their temporary output locations to their final destination on some Storage Element and registers the files in the Replica Location Service of that GRID. - In order to handle the task of ATLAS DC2 an automated production system was designed - The ATLAS production system consists of 4 components: ATLAS production system

26th May ADA model

26th May Ganga in ATLAS Ganga  User interface for job-related operations: configuration, submission, splitting, merging, monitoring, output retrieval, etc LSFGrid 3NorduGridPBSBQSOtherLCG DIAL Production System Condor Other catalogues and repositories DIAL catalogues and repositories ATLAS Metadata Interface (AMI)

26th May Characteristics of DIAL -For Ganga 4, have first concentrated on job submission to DIAL backend -DIAL makes computing facilities available via web services -Server and client functionality implemented in C++  PyDial packaged developed to provide Python bindings to the C++ classes, and functions to simplify their use -A job in DIAL is defined in terms of:  Application: specifies software to be run  Task: specifies configuration information (e.g. job options)  Dataset: specifies data to be processed  Above map to DIAL objects, each with XML representation -Information for applications, tasks and datasets is catalogued

26th May DIAL submission from Ganga (1) -To enable DIAL submission from Ganga, basically needed to implement the following:  Plug-in that uses ADA/DIAL job description  Plug-in that interacts with DIAL backend  Mapping between the two plug-ins -To be able to store DIAL objects in Ganga job repository, needed object-to-XML converters -In Ganga, create instances of DIAL catalogues

26th May DIAL submission from Ganga (2) -From Ganga can do following:  Query the DIAL catalogues  Submit jobs to (remote) DIAL services  Retrieve job output (also partial results)  Via pyROOT, display output histograms  Keep track of jobs from one session to the next  Works well

26th May DIAL submission from Ganga (3) -Current syntax slightly different, but the idea is to be able to do as follows >>> j = AdaJob() >>> j.application = “atlasopt” >>> j.task = “atlas_release jo.py” >>> j.dataset = “hma.dc digit.A1.z_ee.aod files” >>> j.backend.schedulerURL = “lxgate21.cern.ch.20014” >>> j.submit() >>> print j.status >>> print j.result >>> j.copyResult( “myDirectory” ) - Have shown commands given at Python prompt  Same commands can be used in a script

26th May Ganga 4: graphical interfaces -Priority for Ganga 4 has been to have infrastructure in place, and to have functionality available at the command line  Ganga 4 public release planned for June, without GUI -Graphical tools for job configuration and monitoring have been developed for ADA (Alvin Tan)  For jobs using ADA/DIAL job description, expect this to be reused directly in Ganga  Provides a starting point for developing further graphical tools for Ganga

Graphical job builder

Job monitoring

26th May Next steps (1) -Priorities need to be discussed, but there are a number of possibilities  Enable Ganga submission to production system Python tools for entering requests in production database already written (Frederic Brochu)  Enable direct submission from Ganga to one or more Grid flavours  Connect ADA graphical tools to Ganga  Understand interaction with data-management system  Get release out, so that people can try it

26th May Next steps (2) -ATLAS review of distributed analysis tentatively scheduled for July  Must have release for this -Have summer student at Cavendish Laboratory for 6-8 weeks from end of June (Ruth Dixon del Tufo)  Will test usability of Ganga, and help with improvements