1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Ganga 4 Technical Overview Jakub T. Moscicki, CERN

2. Ganga Overview - J.T.Moscicki 2 Enabling Grids for E-sciencE INFSO-RI-508833 Ganga Overview AtlasPR OD DIAL DIRAC LCG2 gLite localhos t LSF submit, kill, get output update status store, retrieve job definition executa ble Gaudi Athena DaVi nci Brun el prepare, configure Ganga4

3. Ganga Overview - J.T.Moscicki 3 Enabling Grids for E-sciencE INFSO-RI-508833 User Interface Ganga.C ore GPI X programming API for job submission/monitoring rapidity/easiness of command line End-User-Interface: enhanced python shell (IPython) Qt-based GUI (3.x) scripts embedding into existing framework: use Ganga as a library/package

4. Ganga Overview - J.T.Moscicki 4 Enabling Grids for E-sciencE INFSO-RI-508833 Ganga Public Interface Ganga GPI Hello World >>> job = Job(exe='/bin/echo',parameters=['hello world']) >>> job.submit() submitting job >>> outfile = file(job.directory+'/output/std.out') >>> print outfile.read() Job started at: Fri Feb 18 14:05:32 2005 Processing input files... /bin/echo Done hello world Application executed with the status code 0 Processing output files... Exiting... Job finished at: Fri Feb 18 14:05:32 2005 >>> job2 = job.copy() >>> job2.backend = “LSF” >>> job2.submit()

5. Ganga Overview - J.T.Moscicki 5 Enabling Grids for E-sciencE INFSO-RI-508833 Ganga Public Interface Inspecting the jobs >>> print job.id 5 >>> print jobs Statistics: 5 jobs jobs -------------- ID status name # 1 completed # 2 new Job20041231647334751371267881024 # 3 completed Job2004123165980541363751081024 # 4 submitted Job2004123182494941363292601024 # 5 completed Job20041231842266811363443961024 >>> for j in jobs[1:3]:... print j.id 1 2

6. Ganga Overview - J.T.Moscicki 6 Enabling Grids for E-sciencE INFSO-RI-508833 Ganga Public Interface Complex scenarios >>> j = Job() >>> j.application = DaVinci() >>> j.application.options = 'my.opts' >>> j.backend = Glite() >>> j.backend.requirements = 'other.GlueCEUniqueID == "grid- ce.desy.de:2119/jobmanager-lcgpbs-short"' more on configuring applications in Atlas and LHCb specific presentations

7. Ganga Overview - J.T.Moscicki 7 Enabling Grids for E-sciencE INFSO-RI-508833 Additional Requirements Can I ? –add a new backend (e.g. PBS) for my site [plug-ins] –add a new application or modify existing one easily [plug-ins] –inspect at home the jobs which I submitted at work [remote job registry] –use Ganga GPI in my python scripts or integrate into existing python system –configure application from my laptop but I haven't installed full ATLAS/LHCB sofwtare release [remote application manager] –make sure that my job output will not be purged from the grid system before I get it [remote job manager]

8. Ganga Overview - J.T.Moscicki 8 Enabling Grids for E-sciencE INFSO-RI-508833 Extending Ganga Creating a Ganga 4 handler is easy –use the plug-in mechanism –example: application handler for Gaudi Create code: –create a class derived from GangaObject –define a schema of data the handler requires –implement the interface for the handler Update the config file ~/.ganga4 [Configuration] RUNTIME_PATH = ~/MyApplication

9. Ganga Overview - J.T.Moscicki 9 Enabling Grids for E-sciencE INFSO-RI-508833 Handler Plugin Example ## Import the GPI from Ganga.GPIDev.Base import GangaObject from Ganga.GPIDev.Schema import * […] class Gaudi(GangaObject): _schema = Schema(Version(1,0),{'optsfile':FileItem(), 'version':SimpleItem(None), 'platform':SimpleItem(None), 'package': SimpleItem(None), 'appname':SimpleItem(None)}) _category='applications‘ Describe the data needed to configure and run an application Set the type of your plugin Create a class based on GangaObject. Provide a Schema for you data Fulfill the interface for the type of handler

10. Ganga Overview - J.T.Moscicki 10 Enabling Grids for E-sciencE INFSO-RI-508833 Handler Plugin Example def configure(self): […] extra=GaudiExtras() […] extra.flatopts=FileParser.writeString(gaudiopts,"exp and") return (None,extra) def list_choices(self,property): class GaudiExtras(GangaObject): _schema = Schema(Version(1,0),{"flatopts": SimpleItem(None)}) _name="GaudiExtras" _category="extras“ _name='Gaudi' Use a GaudiExtras object to return information added by configure Create a Schema with the data of the configured app Set the category of this class to “extras”

11. Ganga Overview - J.T.Moscicki 11 Enabling Grids for E-sciencE INFSO-RI-508833 Remote Job Repository Remote job repository –access to the your jobs from multiple locations –in the future possibility to share jobs between users Ganga Client Ganga Client ARDA MD Server / PostgreSQL local repository

12. Ganga Overview - J.T.Moscicki 12 Enabling Grids for E-sciencE INFSO-RI-508833 Internal architecture Application Manager Job Manager Remote Registry Client Ganga 4 is decomposed into 4 functional components These components also describe the components in a distributed model. Strategy: Design each component so that it could be a separate service. But allow to combine two or more components into a single service

13. Ganga Overview - J.T.Moscicki 13 Enabling Grids for E-sciencE INFSO-RI-508833 Client Application Manager Job Manager Remote Registry Client Runs the Ganga interface (CLIP, GPI, GUI) The user interacts exclusively through the client With the client, the user creates, modifies, submits and monitors jobs Job configuration is kept in a registry which can be local (within the client) or remote.

14. Ganga Overview - J.T.Moscicki 14 Enabling Grids for E-sciencE INFSO-RI-508833 Client Application Manager Job Manager Remote Registry Client The client is a thin client (pure python) The client can be a command line client or a GUI The client interacts with the application manager to configure applications It submits and monitors jobs via the job manager It keeps state by storing persistent information in the registry

15. Ganga Overview - J.T.Moscicki 15 Enabling Grids for E-sciencE INFSO-RI-508833 Application Manager Application Manager Job Manager Remote Registry Client Prepares and configures the application Compiles user code Sets-up the necessary environment Provides information to the client on available applications, versions, platforms, etc.

16. Ganga Overview - J.T.Moscicki 16 Enabling Grids for E-sciencE INFSO-RI-508833 Job Manager Application Manager Job Manager Remote Registry Client Submits the configured job to the submission backend A submission handler submits a job to a backend –Creates the starter script and the JDL – performs the monitoring The application runtime handler –prepares the application dependent wrapper script, depending on the backend. –E.g., DIRAC knows how to run LHCb applications with a different setup as LSF.

17. Ganga Overview - J.T.Moscicki 17 Enabling Grids for E-sciencE INFSO-RI-508833 Remote Registry Application Manager Job Manager Remote Registry Client Keeps track of jobs Is a “passive” data store, typically using a database backend Keeps a roaming profile of the user jobs Keeps track of the job status May cache the output of a job in a local SE. This component is the most likely to be run on a separate host.

18. Ganga Overview - J.T.Moscicki 18 Enabling Grids for E-sciencE INFSO-RI-508833 Ganga Release Status Old series (2.x,3.x) –released regularly in 2004 until March 2005 –integrated wxPython GUI, all basic use-cases implemented for single-client mode and local registry Ganga4 –first public release June, in alpha testing now (alphas released weekly) –completely re-engineered around GPI –integrated LHCb and Atlas/ADA applications –GUI comes later

19. Ganga Overview - J.T.Moscicki 19 Enabling Grids for E-sciencE INFSO-RI-508833 Summary Factsheet (4-0-0-alpha4) –size:  Ganga base: ~300KB, pure-python (no install)  Atlas extensions: ~50KB  LHCb extension: ~50KB –existing functionality:  basic job manipulation via GPI  easy configuration / extension  local and remote registry –planned functionality:  splitting/merging  remote application and job manager  monitoring component

20. Ganga Overview - J.T.Moscicki 20 Enabling Grids for E-sciencE INFSO-RI-508833 http://cern.ch/ganga