1. 2 June 20061/17 http://cern.ch/ganga/ Getting started with Ganga K.Harrison University of Cambridge Tutorial on Distributed Analysis with Ganga CERN, 2 June 2006

2. 2 June 20062/17 Outline -Basics -People/groups involved -Job model -Setting up to run Athena jobs with Ganga -Ganga Command Line Interface in Python (CLIP) -AOD analysis on the Grid -Ganga workspace -Summary

3. 2 June 20063/17 Ganga basics -Ganga is an easy-to-use frontend for job definition and management  Allows simple switching between testing on a local batch system and large-scale data processing on distributed resources (Grid)  Developed in the context of ATLAS and LHCb For ATLAS, have built-in support for applications based on Athena framework, for JobTransforms, and for DQ2 data-management system  Component architecture readily allows extension  Implemented in Python

4. 2 June 20064/17 People/groups involved -Ganga is an ATLAS/LHCb joint project -Support for development work from UK (PPARC/GridPP) and EU (EGEE/ARDA) -Current Ganga contributors:  F.Brochu (Cambridge) U.Egede (Imperial), J.Elmsheuser (München), K.Harrison (Cambridge), D.Liko (CERN), A.Maier (CERN), J.T.Moscicki (CERN), A.Soroko (Oxford), C.L.Tan (Birmingham)

5. 2 June 20065/17 Ganga job abstraction -A job in Ganga is constructed from a set of building blocks, not all required for every job Merger Application Backend Input Dataset Output Dataset Splitter Data read by application Data written by application Rule for dividing into subjobs Rule for combining outputs Where to run What to run Job

6. 2 June 20066/17 Framework for plugin handling -Ganga provides a framework for handling different types of Application, Backend, Dataset, Splitter and Merger, implemented as plugin classes -Each plugin class has its own schema Athena GangaObject IApplication IBackendIDataset ISplitterIMerger LCG -CE -requirements -id -status -reason -actualCE -exitcode -atlas_release -max_events -options -option_file -output -user_area User System Plugin Interfaces Example plugins and schemas

7. 2 June 20067/17 Help with using Ganga -Ganga documentation can be found in the User Guides section of the Ganga web site: http://cern.ch/ganga/http://cern.ch/ganga/  Most relevant items are Installation Working with Ganga (general introduction to functionality) GUI manual (introduction to use of graphical interface) Link to Atlas Wiki page for distributed analysis using Ganga: https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsing Ganga  Today’s tutorial based on this https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsing Ganga -In case of problems or feature requests, do either of the following  Send e-mail to project-ganga@cern.chproject-ganga@cern.ch Must sign up to list to be able to use it  Submit a report via Ganga’s bug-submission page in Savannah: https://savannah.cern.ch/bugs/?func=additem&group=ganga https://savannah.cern.ch/bugs/?func=additem&group=ganga Should either login to Savannah first, or give e-mail address

8. 2 June 20068/17 Setting up to run Athena jobs with Ganga 1. Login to an ATLAS (group zp) account on lxplus - Can work from an account on a different machine, but this implies installing Ganga, an ATLAS release, LCG client and DQ2 tools - Not difficult on a supported platform, but takes time 2. Ensure that you have a Grid certificate installed, and that you are registered with the ATLAS Virtual Organisation 3. Setup environment for Athena, then checkout and build UserAnalysis package 4. Setup the environment for using LCG client 5. Setup the enviornment for DQ2 6. Modify PATH to include directory containing Ganga executable  For detailed instructions, see Wiki page: https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga

9. 2 June 20069/17 Using Ganga -Ganga can be used in any of three ways  Command Line Interface in Python (CLIP) Allows interactive job definition and submission from an enhanced Python shell (IPython) This is good for understanding how the system works (on-line help available), and is used for the first exercises in the tutorial  Scripting This isn’t dealt with explicitly in the tutorial, but any Python/IPython or CLIP commands can be placed in a file and executed in the Ganga environment using the syntax:  Graphical User Interface (GUI) The functionality of the GUI is presented in the last part of the tutorial ganga myScript.py

10. 2 June 200610/17 Running Ganga for the first time 1. Create a Ganga configuration file using: - The configuration file will be created in your home directory, with the name.gangarc 2. Edit ~/.gangarc, which you’ll see includes comments on the configuration possibilities 3. In the section headed [Configuration], add the line - This enables loading of ATLAS-specific plugins 4. Start a Ganga CLIP session by giving the command: - If you don’t have a valid proxy, Ganga will ask for your Grid passphrase ganga -g RUNTIME_PATH = GangaAtlas ganga

11. 2 June 200611/17 Python/IPython commands -All Python/IPython commands can be used at the Ganga prompt -Information about Python can be found at: http://www.python.org/http://www.python.org/  If you’re new to Python, the on-line tutorial is extremely helpful -Information about IPython extensions can be found at: http://ipython.scipy.org/ http://ipython.scipy.org/ -The following are often useful: # A hash (#) marks the start of a comment dir() # List currently available objects help() # Give help help( object ) # Give help on specified object Exit # Exit from session # Use ! before shell commands and use $ before # Python variables to pass to shell, for example: x = 5 !echo $x

12. 2 June 200612/17 Ganga CLIP commands (1) -Ganga commands are explained in the guide Working with Ganga: http://cern.ch/ganga/user/html/GangaIntroduction http://cern.ch/ganga/user/html/GangaIntroduction -From a CLIP session, available commands may be listed, and help can be requested for each -Useful commands include the following list_plugins( “type”) # List plugins of specified type applications() # Alias for list_plugins( “applications” ) backends() # Alias for list_plugins( “backends” ) j1 = Job() # Create a new job j1.application = “Athena” # Set value for application j1.backend = “LCG” # Set value for backend export( j1, “myJob.py” ) # Write job to specified file load( “myJob.py” ) # Load job(s) from specified file j2 = j1.copy() # Create j2 as a copy of job j1 jobs # List jobs

13. 2 June 200613/17 Ganga CLIP commands (2) -When a job j has been defined, the following methods can be used -Once a job has been submitted, it can no longer be modified, and it cannot be resubmitted, but the job can be copied and the copy can be modified/submitted -Ganga supports use of templates, which can be used as the basis of a job definition  The GangaAtlas package defines a template for running Athena on the Local backend that is loaded when you use the package j.submit() # Submit the job j.kill() # Kill the job (if running) j.remove() # Kill the job and delete associated files !ls j.outputdir # List files in job’s output directory templates # List templates j3 = Job( templates[ i ] ) # Create job from template

14. 2 June 200614/17 AOD analysis on the Grid (1) -A job to perform an AOD analysis on LCG can be defined as follows # Create the job j4 = Job( application = “Athena”, backend = “LCG” ) # cd to the run directory of your UserAnalysis package !cd # substitute path to run directory # Prepare user packages for use at remote sites j4.prepare() # Set job-options files, where is full path j4.application.option_file = # Set the input data j4.inputdata = DQ2Dataset( type = “DQ2” ) j4.inputdata.dataset = \ "csc11.005320.PythiaH170wwll.recon.AOD.v11004107” # Set the output files j4.output.files = [ “AnalysisSkeleton.aan.root” ]

15. 2 June 200615/17 AOD analysis on the Grid (2) -Currently very few sites have complete datasets registered with local DQ2 catalogue -To ensure job runs successfully, it’s best to direct it to a site that’s known to have the data available  This is a limitation that will disappear as DQ2 is more widely deployed # Define site where job should run j.backend.CE = \ “ce102.cern.ch:2119/jobmanager-lcglsf-grid_atlas” # Submit the job j.submit()

16. 2 June 200616/17 Ganga workspace -Ganga creates a directory gangadir in your home directory and uses this for storing job-related files and information  You can’t move this directory, but before running Ganga, you can create ~/gangadir as a link to another location gangadir repository input Local templates output workspace Remote gui jobs6667

17. 2 June 200617/17 Summary -The ideas behind Ganga and the syntax of Ganga’s Command Line Interface in Python (CLIP) have been presented in outline -At this point you’re encouraged to try using Ganga to run Athena jobs, following the examples on the Wiki page: https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga https://twiki.cern.ch/twiki/bin/view/Atlas/DistributedAnalysisUsingGanga