1. Karsten Köneke October 22 nd 2007 Ganga User Experience 1/9 Outline: Introduction What are we trying to do? Problems What are the problems? Conclusions Does it work? Ganga User Experience Karsten Köneke DESY

2. Karsten Köneke October 22 nd 2007 Ganga User Experience 2/9 Introduction Introduction Trying to produce common physics ntuples from the AOD: Going to be used by the SUSY Germany group and others. Benefit from a common ntuple basis, i.e. comparable amongst groups. Benefit from and test DESY-HH Tier 2. Our setup: Ganga version 4.4.1 Athena 12.0.6 EventView Group Area 12.0.7.1 Running (mostly) at computing elements FZK, saving to storage element DESY-HH Requirements: Fast and easy large scale production -> GRID! Reliability! Works everywhere, on every site! Easy to use tools for the grid: GANGA

3. Karsten Köneke October 22 nd 2007 Ganga User Experience 3/9 Problems Problems General issues with the grid: Not completely homogeneous by design: Many things can go wrong… Things get changed, sometimes behind the scenes: What the user sees: Script does not work anymore, it used to work last week??? The grid/ganga should maximize the availability of dataset files: See Leonardo’s talk… Ganga now also supports OSG and NorduGrid! (also d.list_locations_num_files() needs to support them…) COMPLETE/INCOMPLETE dataset flags are not always correct! Since ganga chooses by default sites with complete datasets, this needs to be improved/automated How about a global COMPLETE flag if no site alone has a complete copy, but the sum of all sites do…?

4. Karsten Köneke October 22 nd 2007 Ganga User Experience 4/9 Problems Problems Transient issues with the grid: High job failure rate! Some sites ~100%, at good ones about ~10% Sometimes, jobs get “stuck” in a certain state: Currently no solution but to kill and restart! Job completes “successfully”, but no output data is written! Solution: retry New Ganga 4.4.2 puts job in “failed”, so at least user is aware of it! In general, (sub)jobs fail every now and then, even on good sites. Sometimes, a few resubmits are necessary until successful.

5. Karsten Köneke October 22 nd 2007 Ganga User Experience 5/9 Problems Problems A big problem for the inexperienced user: It is sometimes not easy for the user to know what went wrong: User script problem? Ganga problem? Computing element? Storage element? Input files corrupt? DQ2/LCG database? Ganga problem? Problem for user: Who to ask if not sure where the problem is? My solution: Ask in Ganga Hypernews…

6. Karsten Köneke October 22 nd 2007 Ganga User Experience 6/9 Problems Problems User needs to be able to make first diagnosis him/herself: Need tools that access/monitor the running jobs directly! Usable by user to make fast problem diagnostics to actually know what went wrong. There are some in beta status, i.e. this problem has been recognized: New in Ganga 4.4.2: An experimental peek method using Octopus (https://twiki.cern.ch/twiki/bin/view/ArdaGrid/GangaOctopus)https://twiki.cern.ch/twiki/bin/view/ArdaGrid/GangaOctopus Result Monitoring and Online Steering Tool RMOST (Siegen) (http://www.hep.physik.uni-siegen.de/grid/rmost/)http://www.hep.physik.uni-siegen.de/grid/rmost/ Job Monitoring System for LCG JEM (Wuppertal) (http://www.atlas.uni-wuppertal.de/grid/jms/)http://www.atlas.uni-wuppertal.de/grid/jms/ Seems very promising… Let’s see what the user experience with these new tools are…

7. Karsten Köneke October 22 nd 2007 Ganga User Experience 7/9 Evolution Evolution The Ganga developers: Over the last two months (or so, since I started): Response of ganga developers was outstanding!!! A lot of feedback/requests got implemented very fast!!! -> 5 versions of ganga in that time! user feedbackSymbiosis of developers and user feedback, e.g: Multiple environment variables with gLite File and DQ2 dataset naming more user specific File merging problem resolved Open issues, e.g: Resubmitting failed subjobs with gLite: Expected in Ganga 4.4.3 or 4.4.4. Automated resubmission? Max retry count to be set in submit script? To the Ganga team: KEEP UP THE EXCELLENT WORK!!

8. Karsten Köneke October 22 nd 2007 Ganga User Experience 8/9 Suggestions for first time users Suggestions for first time users Subscribe to these Hypernews forums: https://hypernews.cern.ch/HyperNews/Atlas/get/GANGAUserDeveloper.html All kinds of help, or at least they know whom to ask… https://hypernews.cern.ch/HyperNews/Atlas/get/gridAnnounce.html Here, issues with sites get announced. Take a look if something is not working where you think it should work, maybe the site has a problem?! Follow the current tutorial: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorial44 A good twiki with lots of working examples for all sorts of things (p.s.: The link from the ganga homepage still links to 43) Ganga homepage: http://ganga.web.cern.ch/ganga/user/index.php When you find a problem: REPORT IT!

9. Karsten Köneke October 22 nd 2007 Ganga User Experience 9/9 Conclusions Conclusions Ganga is a very nice tool!!! And it is (despite my complaining earlier) stable! There are still some hiccups and room for improvement… Ganga developers are very active and responsive!!! The grid is a great idea, but: since things are very spread out, difficult to track down and solve problems for the simple user. Users need to use ganga/grid now and give feedback now! It is too late when the real data comes and everybody wants to run!Users need to use ganga/grid now and give feedback now! It is too late when the real data comes and everybody wants to run!