1. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 1 EnablingGrids for E-siencE Generic Application: Monte Carlo Production for the MAGIC telescope MAGIC Grid Status report Harald Kornmayer, Alberto Forti, Toni Coarasa Mirco Mazzucato, Manuel Delfino Forschungszentrum Karlsruhe; University of Udine; Max-Planck-Institut für Physik, München; CNAF/INFN Bologna; PIC, Bologna on behalf of the MAGIC collaboration

2. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 2 EnablingGrids for E-siencE Outline Introduction What kind of MAGIC? Development Use CasesRequirements WorkflowsMetaData Status HardwareSoftware Deployment Experience First Month on production is over Grid-added value

3. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 3 EnablingGrids for E-siencE Introduction: The MAGIC Telescope Ground based Air Cerenkov Telescope Gamma ray: 30 GeV - TeV LaPalma, Canary Islands ( 28° North, 18° West ) 17 m diameter operation since autumn 2003 (still in commissioning) Collaborators: IFAE Barcelona, UAB Barcelona, Humboldt U. Berlin, UC Davis, U. Lodz, UC Madrid, MPI München, INFN / U. Padova, U. Potchefstrom, INFN / U. Siena, Tuorla Observatory, INFN / U. Udine, U. Würzburg, Yerevan Physics Inst., ETH Zürich Physics Goals: Origin of VHE Gamma rays Active Galactic Nuclei Supernova Remnants Unidentified EGRET sources Gamma Ray Burst

4. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 4 EnablingGrids for E-siencE ~ 10 km Particle shower Ground based γ-ray astronomy ~ 1 o Cherenkov light ~ 120 m Gamma ray GLAST (~ 1 m 2 ) Cherenkov light Image of particle shower in telescope camera reconstruct: arrival direction, energy reject hadron background

5. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 5 EnablingGrids for E-siencE MAGIC – Hadron rejection Based on extensive Monte Carlo Simulation air shower simulation program CORSIKA Simulation of hadronic background is very CPU consuming to simulate the background of one night, 70 CPUs (P4 2GHz) needs to run 19200 days to simulate the gamma events of one night for a Crab like source takes 288 days. At higher energies (> 70 GeV) observations are possible already by On-Off method (This reduces the On-time by a factor of two) Lowering the threshold of the MAGIC telescope requires new methods based on Monte Carlo Simulations

6. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 6 EnablingGrids for E-siencE Developments - Requirements MAGIC needs a lot of CPU to simulate the hadronic background to explore the energy range 10 GeV – 100 GeV MAGIC needs a coordinated effort for the MonteCarlo production MAGIC needs an easy accessible system (Where are the data from run_1002 and run_1003?) MAGIC needs an scalable system (as MAGIC II will come 2007) MAGIC needs the possiblity to access data from other experiments (HESS, Vertias, GLAST, PLANCK(?)) for multiwavelength campaigns

7. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 7 EnablingGrids for E-siencE Development - Team The proposal was written in March 2004 Support for the proposal from –CNAF (M. Mazzucato) –PIC (M. Delfino) –FZK –MAGIC Organisation –Contact Person Harald Kornmayer –VO Manager Harald Kornmayer –Gridifying developers Alberto Forti Harald Kornmayer Toni Coarasa

8. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 8 EnablingGrids for E-siencE Development – MC Workflow I need 1.5 million hadronic showers with Energy E, Direction (theta, phi),... As background sample for observation of „Crab nebula“ Run Magic MonteCarlo Simulation and register output data Run Magic Monte Carlo Simulation (MMCS) and register output data Simulate the Telescope Geometry with the reflector program for all interesting MMCS files and register output data Simulate the Starlight Background for a given position in the sky and register output data Simulate the response of the MAGIC camera for all interesting reflector files and register output data Merge the shower simulation and the StarLight simulation and produce a MonteCarlo data sample

9. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 9 EnablingGrids for E-siencE Huge number of runs with different parameters  We want to record what we did!  MetaData needed! MMCS – Meta data Model for MMCS Store run parameters Monitor Grid Jobs Manage MMCS data for reflector prototype designed for camera need to be done!

10. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 10 EnablingGrids for E-siencE Status - Hardware History: –The Hague, Nov. 2004: The MMCS prototype was developed on the CrossGrid testbed First test on GILDA Migration to real EGEE infrastructure needed Now: –Running on real EGEE infrastructure (april 2005) VO server at Nihkef/SARA BDII and Resource Broker at CNAF Replica services at PIC Resources (whereever available!  come to that later!) bash-2.05a$ lcg-infosites --vo magic all LRC endpoint for magic: http://rlsmagic.pic.es:8080/magic/edg-local-replica-catalog/services/edg-local-replica-catalog RMC endpoint for magic: http://rlsmagic.pic.es:8080/magic/edg-replica-metadata-catalog/services/edg-replica-metadata-catalog LRC endpoint for magic: http://rlsmagic.pic.es:8080/magic/edg-local-replica-catalog/services/edg-local-replica-catalog RMC endpoint for magic: http://rlsmagic.pic.es:8080/magic/edg-replica-metadata-catalog/services/edg-replica-metadata-catalog **************************************************************** These are the related data for magic: (in terms of CPUs) **************************************************************** #CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------- 20 20 0 0 0 ce01.pic.es:2119/jobmanager-torque-magic 168 4 1 1 0 lcgce02.ifae.es:2119/jobmanager-lcgpbs-magic 7 6 0 0 0 ce001.grid.bas.bg:2119/jobmanager-lcgpbs-magic 82 63 6 6 0 gridba2.ba.infn.it:2119/jobmanager-lcgpbs-long 82 63 3 3 0 gridba2.ba.infn.it:2119/jobmanager-lcgpbs-short 1280 5 0 0 0 lcg-gridka-ce.fzk.de:2119/jobmanager-pbspro-long 1280 5 0 0 0 lcg-gridka-ce.fzk.de:2119/jobmanager-pbspro-short 50 29 1 1 0 grid012.ct.infn.it:2119/jobmanager-lcgpbs-long 50 29 0 0 0 grid012.ct.infn.it:2119/jobmanager-lcgpbs-short 82 63 8 8 0 gridba2.ba.infn.it:2119/jobmanager-lcgpbs-infinite 1280 5 0 0 0 lcg-gridka-ce.fzk.de:2119/jobmanager-pbspro-default 582 413 0 0 0 ce01-lcg.cr.cnaf.infn.it:2119/jobmanager-lcglsf-magic 1280 5 0 0 0 lcg-gridka-ce.fzk.de:2119/jobmanager-pbspro-extralong 50 29 24 20 4 grid012.ct.infn.it:2119/jobmanager-lcgpbs-infinite 8 6 1 1 0 gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-long 8 6 0 0 0 gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-short 8 6 1 1 0 gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-infinite ---------------------------------------------------------- The total values are: ---------------------------------------------------------- 4889 648 45 41 4 ************************************************************** These are the related data for magic: (in terms of SE) ************************************************************** Access point Avail Space(Kb) Used Space(Kb) Files SEs --------------------------------------------------------------------------------------------------------------- gsiftp://castorgrid.ifae.es/castor/ifae.es/lcg/magic/ 1000000000000 0 0 castorgrid.ifae.es gsiftp://castorgrid.pic.es/castor/ifae.es/lcg/magic/ 1000000000000 0 0 castorgrid.pic.es gsiftp://castorgrid.ifae.es/castor/ifae.es/lcg/magic/ 53020932 0 0 castorgrid.ifae.es gsiftp://se001.grid.bas.bg/storage/magic/ 354813480 30087 4 se001.grid.bas.bg gsiftp://gridba6.ba.infn.it/flatfiles/SE00/magic/ 1605498664 0 0 gridba6.ba.infn.it gsiftp://grid007g.cnaf.infn.it/flatfiles/SE00/magic/ 1497769448 0 0 grid007g.cnaf.infn.it gsiftp://lcg-gridka-se.fzk.de/grid/fzk.de/..../SE00/magic/ 1605498664 0 0 lcg-gridka-se.fzk.de

11. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 11 EnablingGrids for E-siencE Status – Hardware Nikhef/SaraFZK BA.INFN.IT PICCNAF BAS.BG BDIIMetaDataBase UI VO-Server UI RB CE UICE RLS SECE MPI.DE UICE CT.INFN.IT CE This system is running and used for the first Monte Carlo Production challenge on the Grid (12 MEvts)  First experience EGEE-1 CE EGEE-2 CE EGEE-2 CE (apr/may 2005) SE

12. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 12 EnablingGrids for E-siencE Status – Software MMCS system version 1.0 is up and running! Reflector prototype tested Camera needs to be done! Currently we submit all the software with the jdl  no software installation required yet  methods like the GAIT are tested (see Crakow Grid Workshop 2004) The submission needs approx. 40 sec With a more dedicate application installation procedure, this can be improved!! MMCS system version 1.0 Java based prototype Hides the EDG/lcg command from the user Access the Meta-database Runs on User Interface

13. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 13 EnablingGrids for E-siencE Experience – Execution Data challenge Grid-1 12M hadron events 12000 jobs needed started march 2005 up to now ~ 4000 jobs First tests: with manual GUI submission Reasons for failure Network problems RB problems Queue problems Job successful: Output file registered at PIC Diagnostic: no tools found complex and time consuming  use metadata base, log the failure, resubmit and don‘t care 170/3780 Jobs failed  4.5 % failure

14. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 14 EnablingGrids for E-siencE Experience – InputProcOutput How to submit a job?  JDL JDL should specify Get the input form InputSandBox InputData Run the program Executable Store the output at OutputSandBox OutputData InputData OutputData File on UI InputSandBox File on Grid InputData File to UI OutputSandBox File to Grid OutputData OK! No file transfer! OK! Answer from experts: write a script that copy the file a SE to the WN BUT: I don‘t want to implement a WORKAROUND for basic grid functionality

15. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 15 EnablingGrids for E-siencE The VO is managed with the VO server Why do the MAGIC grid users needs to be registered at CERN? Clarification of the non-LCG-registration procedure needed!! SA1 centers must know about this procedure!! VO manager has no means to rejected offered resources Only the local sites decide, if they want to support the VO There is no certification/test procedure before  this can produce instability of a running production system! The VO manager can not remove them! Who is managing the VO? Experience – VO Management

16. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 16 EnablingGrids for E-siencE Qualitative Enable „(e-)scientific“ collaboration between partners Enable the cooperation between different experiments „Towards a virtual observatory for VHE gamma rays“ (presentation next week) Enable the participation on Virtual Observatories Grid added value Quantitative Number of temporarily available CPU unreachable no money, no manpower Aim: 1.5 Mio Events in 24 hours Manpower for storage maintenance reduced

17. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 17 EnablingGrids for E-siencE Technical: „missing“ functionality in middleware IPO definition Job status information to computing centric VO management Set up Local mapping Central registration bottlenecks Technical: Tools for error diagnostics not available VO supporting tools More Batch systems need support Sun Grid Engine Firewall The range of open firewall ports is a problem for new computing centers

18. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 18 EnablingGrids for E-siencE Political: … bottlenecks Money: Near-future availablility EGEE-2 resources (?) Beyond 2007 (?) Far-future Grid operation models? Sustainablility ? Manpower: EGEE Account Manager Administrative: Single point of contact  VO account manager Many boards EGAAP vs. OAG Many documents read Where to Send Middleware Requirements? Communication flow inside EGEE not obvious for new VOs

19. H. Kornmayer MAGIC-GRID Status report EGAAP meeting, Athens, 21th April 2005 - 19 EnablingGrids for E-siencE EGEE – MAGIC Grid From TheHague to Athens MAGIC Grid on EGEE became real First Monte Carlo Production challenge 12 Mio Events started mid of March ongoing Evaluation of a Grid based analysis system for MAGIC New application domain on the Grid „Towards a virtual VHE gamma ray observatory“  wait for the future