Ganga 4 Basics - Tutorial Jakub T. Moscicki ARDA/LHCb Ganga Tutorial, September 2006

Ganga Tutorial, April Contents What is Ganga  7 things to know about Ganga from K.Harrison Setup and running for the first time Quick introduction to Python Simple Operations in Interactive Text Shell Copying jobs and submitting to other backends Scripting interface Shipping files Splitting

Ganga Tutorial, April What is Ganga? AtlasPROD DIAL DIRAC LCG2 gLite localhost LSF submit, kill get output update status store & retrieve job definition prepare, configure Ganga4 Job scripts Gaudi Athena AtlasPROD DIAL DIRAC LCG2 gLite localhost LSF

Ganga Tutorial, April Ganga Web Site:  Ganga Tutorial Bug reports Ganga is based on Python 2.2  Python Website:  Excellent Python Tutorial: Other resources:  IPython page: Documentation

Ganga Tutorial, April Setting up Ganga ATLAS tutorial instructions! Note: we are using a beta version – new release by the end of this month % ganga *** Welcome to Ganga *** Version: Ganga beta7 Documentation and support: Type help() or help('index') for online help. You will be asked few questions. Answer “yes” by pressing ENTER. GRID certificate: type your password or press Control-D several times. To quit press Control-D at the interactive prompt In[1]: ^D Do you really want to exit ([y]/n)? %

Ganga Tutorial, April Ganga Text Prompt Ganga is based on python 2.2 and has an enhanced python prompt (IPython):  python programming/scripting var = 5 print var*10  easy access to the shell commands !less ~/.gangarc # personal config file !pwd  history  TAB completion it works on keywords, variables, objects, try: –va  many more features

Ganga Tutorial, April x = 2 print x*3 # you may also skip print if x==2: print "yes, x==2" # NOTE INITIAL SPACES! alist = [1,2,3] for y in alist: print y print len(alist) # built-in function len help(len) Python: Statements, Variables

Ganga Tutorial, April print range(11) help(range) def square(v): return v*v print square(x) alist = [1,2,3] for i in range(len(alist)): alist[i] = square(alist[i]) blist = [square(x) for x in alist] Python: Functions

Ganga Tutorial, April import sys print sys.argv import os print os.environ['HOME'] from os import environ from os import * import math, cmath print math.cos(math.pi) v =(1+1j) print v*v math.log(-1) # exception Python: Modules

Ganga Tutorial, April # standard library & user classes import random r = random.Random() r.choice([1,2,3,4]) r.randint(30,40) #built-in classes: for example strings s = "hello at the tutorial" # double quotes s.split() s.count('a') s.upper() dir(s) if 'tut' in s: print 'tut is in ',s print '-'*20 # single quotes Python: Classes and Objects

Ganga Tutorial, April # file objects f = file('/etc/password') lines = f.readlines() print len(lines) # writing files f2 = file('testfile','w') print >> f2, “this is a test”,20*'-' f2.close() !cat testfile NOTE: Ganga defines a special class File. Do not confuse File and file! Python: files

Ganga Tutorial, April Job Objects and Job Registry [In:] jobs [In:] Job() [In:] jobs [In:] j = jobs[1] [In:] print j.application [In:] print j.backend # job registry: a list of all your jobs [In:] jobs [In:] jobs[1] Ganga Scripting

Ganga Tutorial, April Jobs Job ( status = 'new', inputdir = '/afs/cern.ch/user/k/kuba/gangadir/workspace/Local/2/input/', name = '', outputdir = '/afs/cern.ch/user/k/kuba/gangadir/workspace/Local/2/output/', outputsandbox = [], id = 2, application = Executable ( exe = '/bin/hostname', env = {}, args = [] ), inputdata = None, inputsandbox = [ ], backend = Local ( exitcode = None, id = None )

Ganga Tutorial, April [In:] j = Job() [In:] j.application.exe = '/bin/hostname' [In:] j.name = "MyTest" [In:] print j [In:] j.submit() # wait until job is completed and look # at the output directory [In:] print j.status [In:] print j.outputdir [In:] !ls $j.outputdir [In:] !cat $j.outputdir/stdout # the ! and $ syntax is IPython – you # cannot do it in normal python # it is useful for interactive work # instead of mouse copy-paste Basic Job Submission

Ganga Tutorial, April # once a job is submitted you cannot modify # it. if you want to submit a new job you # should create a new job object [In:] j2 = j.copy() [In:] j2.backend = LSF() [In:] j2.submit() # if you have GRID certificate you can try [In:] j3 = j.copy() [In:] j3.backend = LCG() [In:] j3.submit() # print jobs to see all your jobs More submission

Ganga Tutorial, April # see predefined plugins # (only some combinations supported) list_plugins() help('index') # submission in a loop for i in range(5): j = Job() j.application.exe="/bin/echo" j.application.args=["hello ",str(i)]) print j.id, j.inputdir print j.submit() More ideas

Ganga Tutorial, April # For frequently used jobs – use templates! t = JobTemplate(j) print templates j = Job(t) # Create a new job from t # Other useful methods j.kill() j.remove() # Removes the output as well! for j in jobs: j.kill() jobs.clean() # Faster! Deletes ALL jobs. More Ideas (2)

Ganga Tutorial, April export(j,'myjob') # press ^D to quit ganga cat myjob ganga submit myjob ganga query *restart ganga* load('myjob') Scripting possibilities

Ganga Tutorial, April j = Job() j.application.exe=File('spy') j.outputsandbox = ['*.txt'] j.submit() Do not confuse File with file! Shipping files

Ganga Tutorial, April Prime number crunching:  cryptography (also Grid certificates)  you break the key by factoring large integers into prime numbers Application setup:  we have 15 tables with 1M primes each the files are downloaded from a web server  we have a “prime_factor.py” script which scans the table and returns the factors it finds: [(p1,n1),(p2,n2),...,(pk,nk)] so that product of pow(pi,ni) = the number Splitting Demo

Ganga Tutorial, April Job 1:  find all primes factors of among the first million primes j = Job() j.application.exe=File('prime_factor.py') j.application.args =[' ', ' j.outputsandbox=['*.dat'] j.submit() cat $j.outputdir/*.dat execfile('primes.py') # load helper functions for primes check_prime_job(j) Splitting Demo

Ganga Tutorial, April Job 2: find more factors using 5 tables j = j.copy() split_prime_job(j,5) Exercise:  find all primes of: Notes:  you cannot not crack real codes this way ;-)  instead of the Web server one could/should use a Grid SE Splitting Demo

Ganga Tutorial, April Try: ganga -h to see options Have a look also in the config file: ~/.gangarc Hints and Pitfalls

Ganga Tutorial, April Questions? Practical exercises Questions?