1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Large Simulations using EGEE Grid for the Pierre Auger Observatory Jiří Chudoba 1,2, Jaroslava Schovancová 1,2, Petr Trávníček 1 1) Institute of Physics, AS CR and 2) CESNET 8.2.2008, ASPERA meeting, Paris

2. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 2 Pierre Auger Collaboration More than 250 PhD scientists from more than 60 institutions from 16 (+2) countries > 400 authors Participating countries: Argentina, Australia, Bolivia *, Brazil, Czech Republic, France, Germany, Italy, Mexico, Netherlands, Poland, Portugal, Slovenia, Spain, United Kingdom, USA and Vietnam * * - associated countries

3. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 3 Pierre Auger Observatory Southern observatory almost completed Some results already presented Southern hemisphere: Malargüe, Mendoza province, Argentina Northern hemisphere (planned): Lamar, Colorado, USA Lifetime of the observatory: 15 - 20 yrs

4. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 4 Auger Computing Model Data recorded at the experiment site are copied to Lyon - CCIN2P3 –slow line from Malargue, transfer via recorded DVD –EU funded project AUGERACCESS will improve connectivity All Auger members have accounts at CCIN2P3 Data is analyzed remotely at CCIN2P3 or copied and processed on PCs or small computing farms –users must install new versions of the offline sw –users check for availability of new data CPU intensive simulations done at various sites and results copied to CCIN2P3 –some sets are not at CCIN2P3, available to only some users

5. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 5 Expectations from Grid more computing resources available to more collaboration members uniform way for simulation productions easier access to data tests with simulations were done using regional VOs (central Europe - VOCE, SouthWest)

6. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 6 VO AUGER Created in 2006 by the Czech group in cooperation with CESNET (Czech EGEE partner) CESNET provides and maintains central services –LCG RB, gLite WMS, LB, UI, LFC, registration portal, VOMS server, JP server VOMS server replica at FNAL 10 sites from 8 countries support VO AUGER EELA project considers AUGER as supported application –E-infrastructure shared between Europe and Latin America –www.eu-eela.org, IST-2006-026409www.eu-eela.org –UNAM (National Autonomous University of Mexico) is the 1 st EELA site supporting VO AUGER

7. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 7 Registration to the VO AUGER

8. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 8 Resources, Usage Production of libraries of cosmic ray showers Access to the simulated data files Resources for individual users golias25.farm.particle.czPrague, ASCR156 TB skurut17.cesnet.czPrague, CESNET201 TB apcpc79.in2p3.frGRIF (APC), Paris50 TB grid10.lal.in2p3.frGRIF (LAL), Paris150 TB ce02.lip.ptLIP, Lisboa20 TB tbn20.nikhef.nlNIKHEF, Amsterodam200 TB lcgce.ijs.siIJS, Ljubljana102 TB UNAM40 TB CCIN2P3, Lyon FZK, Karlsruhe10050 TB Aachen8515 TB Wuppertal5025 TB Number of CPUs and available storage for the VO Auger are rough estimates !! They fluctuate by a large factor

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 9 CPU Challenge in April 2007 –number of really usable CPUs varies

10. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 10 Corsika simulations Offline software: CORSIKA http://www-ik.fzk.de/corsika/ –(v. 6.611 for tests, v. 6.617 for proton and iron production) Model: –epos, fluka Primary particle –proton (epos_gr01, 3, 7) or iron (epos_gr04, 6) Energy of primary particle –log(E/eV)=17.5, 18.0, 18.5, 19.0, 19.5, 20.0, 20.5, 21.0 Zenith angle –q=0, 18, 26, 38, 45, 53, 60 deg 50 jobs per each [E, q] pair –different random number seed –seed generated using srand(time(NULL)) and rand() functions implemented in standard math.h library

11. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 11 CORSIKA Parameters

12. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 12 Shower libraries epos_gr01proton, EPOS1 – 2800, test (deleted) epos_gr02proton, EPOStest (deleted) epos_gr03proton, EPOS1 - 2800 epos_gr04iron, EPOS1 - 2800 epos_gr05proton, EPOSJPtest epos_gr06iron, EPOS2801 - 5600 epos_gr07proton, EPOS2801 - 5600

13. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 13 Library productions 2800 jobs per production Test production (epos_gr01) –August 28 – September 10, 2007 (14 days) 1 st usable proton production (epos_gr03) –October 3 – 24, 2007 (major part in 21 days) –2745 jobs finished, < 2.5 % crashes 1 st iron production (epos_gr03) –October 22 – December 15, 2007 –2200 jobs finished, < 2.5 % crashes Job outputs stored on SE –Primarily at golias100.farm.particle.cz, later also at dcache.ijs.si

14. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 14 Running jobs – proton showers CE epos_gr03 (proton) CE JobsCPU timeAvg. CPU [#][hr]time [hr] ce02.lip.ptLIP, Lisbon4569715 golias25.farm.particle.czFZU, Prague397987525 grid10.lal.in2p3.fr+apcpc79GRIF, Paris9187610 lcgce.ijs.siIJS, Ljublan6181239020 skurut17.cesnet.czCESNET, Prague140606443 tbn20.nikhef.nlNIKHEF, Amsterdam14701544811 JOBS COMPUTED (DONE) 27614535116 SHOWERS DONE 25954261316 TOTAL (SHOWERS) 2800

15. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 15 Running jobs – iron showers CE epos_gr04 (iron) JobsCPU timeAvg. CPU [#][hr]time [hr] ce02.lip.ptLIP, Lisbon65115618 golias25.farm.particle.czFZU, Prague9372938231 grid10.lal.in2p3.fr + apcpc79GRIF, Paris143176712 lcgce.ijs.siIJS, Ljublan447998122 skurut17.cesnet.czCESNET, Prague91790087 tbn20.nikhef.nlNIKHEF, Amsterdam18072499514 JOBS COMPUTED (DONE) 34907518122 SHOWERS DONE 27285301119 TOTAL (SHOWERS) 2800 example of a logical file name: lfn:/grid/auger/prod/epos_gr04/en20.000/th18.000/DAT001845.part

16. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 16 Site ratios

17. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 17 Walltime per site

18. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 18 Used walltime per library

19. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 19 Summary production of CORSIKA libraries for AUGER files are accessible by VO auger members plans to copy (part of) libraries to CCIN2P3 current productions limited by available storage working on the detector simulation on the grid expect at least double CPU capacity soon challenge to keep these resources busy

20. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 20 Backup slides

21. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 21 gr06, gr07 CE epos_gr06 (iron) Jobs CPU time Avg. CPU [#][hr]time [hr] ce02.lip.ptLIP, Lisbon2648719 golias25.farm.particle.czFZU, Prague84394647 gridka...FZK, Karlsruhe00-- grid10.lal.in2p3.frLAL, Paris7511086114 lcgce.ijs.siIJS, Ljublan44104424 skurut17.cesnet.cz CESNET, Prague11221462213 tbn20.nikhef.nl NIKHEF, Amsterdam758930612 JOBS COMPUTED (DONE) 27854026514 SHOWERS DONE 26783879114 TOTAL (SHOWERS) 2800 CE epos_gr07 (proton) Jobs CPU time Avg. CPU [#][hr]time [hr] ce02.lip.ptLIP, Lisbon3948112 golias25.farm.particle.czFZU, Prague45156935 gridka...FZK, Karlsruhe17 1 grid10.lal.in2p3.frLAL, Paris14251847313 lcgce.ijs.siIJS, Ljublan1531035968 skurut17.cesnet.czCESNET, Prague977428 tbn20.nikhef.nl NIKHEF, Amsterdam244294612 JOBS COMPUTED (DONE) 20203458617 SHOWERS DONE 125209317 TOTAL (SHOWERS) 2800

22. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 22 Jobs per day

23. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 ASPERA meeting, Paris 2008 23 Walltime