1. EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks C. Loomis (CNRS/LAL) NA4 Activity Manager EGEE-III First Review, 24-25 June, 2009 NA4: User Community Support and Expansion

2. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Activity Overview NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 2 CountryPMFTE Austria60.3 Belgium120.5 CERN1626.8 Cyprus120.5 Czech Republic 170.7 France25710.7 Germany472.0 Greece923.8 Hungary1265.3 Israel120.5 Italy2098.7 Netherlands291.2 Norway301.3 Poland421.8 Russia662.8 Slovakia241.0 Spain2118.8 Sweden160.7 UK411.7 TOTAL142159.2 44Partners 287People 19Countries NA4: 19%

3. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Tasks TNA4.1: Support –Virtual Organization Support (VOS) –Application Porting Support (APS) –Direct User Support (DUS) TNA4.2: Strategic Discipline Clusters –High Energy Physics (HEP) –Life Sciences (LS) –Earth Sciences (ES) –Grid Observatory (GO, CS) –Computational Chemistry (CC) –Astronomy & Astrophysics (AA) –Fusion (F) TNA4.3: Activity Coordination –Activity Management –Regional Coordination NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 3

4. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 User Community Organization NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 4 User VO Domain User Community Grid Auth. Clusters

5. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Community & Use NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 5 DomainVOsUsers AA20373 CC4347 CS421 ES7142 F268 HEP368577 LS9379 MV261658 OTH281816 TOTAL13613381 Around 13000 Registered Users Consistent doubling every 12-18 months. EGEE-III = Y2EGEE-II = Y1 Accounting Portal: http://www3.egee.cesga.es/http://www3.egee.cesga.es/ CIC Portal: http://cic.gridops.org/http://cic.gridops.org/

6. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 CPU Utilization by Domain Domain VOs (>0%) VOs (>10%) VO Names (>10%) AA113astro.vo.eu-egee.org, auger, virgo CC42compchem, trgrida CS21imath.cesga.es ES41esr F11fusion HEP324alice, atlas, cms, lhcb LS71biomed MV125aegis, balticgrid, see, seegrid, vo.gear.cern.ch OTH192geant4, theophys UNK793bg, litgrid, vo.nanocmos.ac.uk NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 6 112 Registered VOs 171 “Visible” VOs 23 “Core” VOs 4167 “Core” Users DomainY1Y2Y2/Y1 AA2520157266.2 CC17400292061.7 CS15242.6 ES286330111.5 F368114280.4 HEP2027944443992.2 LS14343196981.4 MV12797241641.9 OTH3013145284.8 UNK413555241.3 TOTAL2609705580252.1 2x increase overall HEP largest users / contributors AA/ES/OTH show strong increase CPU Use: 1K-SI2K-Month

7. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Applications NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 7 http://appdb.eu-egee.org Alt. link: http://grid.ct.infn.it/egee_applications/

8. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Virtual Organization Support VO Management Developments –Improving the VO registration information –Integration of collaborative tools with VO information –Expansion of SAM testing framework for non-LHC VOs Documentation and Support Provision –Links to VO documentation: https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport –Liaison between operations and VO managers VO Tools Identification –Identified problems with VOMS functionality –Worked in collaboration with JSPG on changes to policies related to VO management NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 8

9. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Application Porting Support Consultancy and Porting –15 applications ported; ~10 applications being ported Training –Group collects, prepares, reorganizes training materials and offers those as customized training packages for users –Direct participation in NA3 training events Provision of Infrastructure Services –Group leader is VO manager for the NA4 –Partly responsible for Application Database Public Relations –Writing of success stories of ported applications to increase visibility and to help others with similar applications –“EGEE App. Porting Support Group” won Best Demo prize at EGEE’08 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 9

10. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 http://www.ldps.sztaki.hu/gasuc/ Application Porting Support NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 10 10 applications being ported 15 applications ported

11. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Direct User Support Ticket Handling –DUS support unit part of GGUS since mid-September –Have taken 2-person, 2-week shifts to treat tickets –The number of tickets assigned to DUS has been small Documentation and Use Cases –Reviewed and accessed existing documentation –Writing new documentation to fill identified gaps –Working with clusters and other teams to improve their documentation NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 11

12. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Ganga/DIANE, AMGA, Dashboard –Used by 1000s in HEP, strong adoption by other communities –Ganga/DIANE tutorials: NSS IEEE, Helsinki, BalticGrid –Dashboard tutorial: UF4/OGF25 –CERN Training for Trainers Grid validation for LHC data taking: CCRC’08, STEP’09 –4 expts., 3 grid infra., 100s of sites, O(PB) data –Sustained 4GB/s CERN  T1, O(100K) jobs/day High Energy Physics NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 12

13. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Technology Transfer Collab. HEP and Fusion clusters (since EGEE’08) –Porting of specific fusion applications using Ganga/DIANE –Results of the collaboration shown during UF4/OGF25 Lattice QCD (in production since 2007) –Running autonomously on a daily basis using Ganga/DIANE –Sustained rate of 1000 concurrent jobs, 750 CPUs and more than 20TB of data transferred GEANT4 simulation toolkit (since 2005) –Widely used by astroparticle physics, medical applications, radiation studies, as well as HEP –Validation of the new releases performed regularly on EGEE grid, again using Ganga/DIANE NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 13

14. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Life Sciences Major calculations: –WISDOM (http://wisdom.eu-egee.fr/)http://wisdom.eu-egee.fr/ –System biology on cancer data –Genetic linkage analysis for disease loci –Identification of causes for coronary artery diseases Tooling Support –AMGA –Medical Data Manager –MOTEUR –Taverna2 Plug-In NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 14

15. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Nature Genetics Article Genome-wide haplotype analyses of complex human diseases –Study the impact of DNA mutations on human coronary diseases –Very CPU intensive analysis to study the impact of correlated (double, triple) DNA mutations EGEE grid deployment –1926 Coronary Artery Disease patients; 2938 healthy controls –378,000 Single Nucleon Polymorphisms = local DNA mutations –8.1 million combinations tested in less than 45 days (instead of >10 years on a single Pentium 4) Results in Nature Genetics Mar. 2009 (D. Tregouet et al) –Major role of mutations on chromosome 6 was confirmed. NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 15

16. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Earth Sciences User and application support Dissemination –Session at European Geosciences Union (EGU) in 2008 –Special issue of journal with 12 peer-reviewed papers –2 PhDs and 5 papers based on Geocluster results from EGEE Specialized tools –Data distribution, file explorer, storage access systems, workflow tools, … NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 16

17. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Pesticide risk assessment and management in Europe –FP6 EU project –BRGM France + 14 partners in 9 countries Creation of a large database including 4 million scenarios (climate, soil, pesticides, …). Successful results with the first 2 million scenarios obtained with EGEE running around 4800 jobs/day. Exploitation of database and results by all partners. –Creation of one SME in France for agriculture consultancy. Footprint NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 17

18. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Grid Observatory Created the Grid Observatory Portal –Store and publish monitoring information for analysis Reaching out to CS community: –EGEE’08: Grid Community Meeting –UF4/OGF25: Joint session “From Grid Monitoring to Analysis” –Grid Meeting Autonomic Computing (GMAC’09) at ICAC NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 18

19. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Grid Observatory Portal NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 19 http://grid-observatory.org/

20. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Computational Chemistry Analysis of grid licensing models. Expanding membership –Training of young researchers –Availability of necessary software packages Tooling: –Chempo, Charon, ECCE, Wien2K –Parallel version of GAMESS NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 20

21. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Computational Campaigns Chemical reactions –N + N 2, O + O 2 and F + HD –Thermal rate coefficients Nanotube modeling P-Grade port of ABC program NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 21 Executor: executed as many times in parallel as many parameters are generated by “Generator” Collector: collects all output files into a single TAR file Generator: generates input files with different parameters (currently 4 input)

22. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Astronomy & Astrophysics Development of active community –Large number of applications ported to the grid –Focused training and dissemination Tooling: –Management of parameter sweep applications –Scheduling and bookkeeping systems –Visualization Interaction with EuroVO NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 22

23. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Planck Satellite Launched 14 May will be in L2 orbit in early July. INAF: Ported full LFI mission simulation to EGEE IFCA: Ported several codes to LFI Data Processing Center: –Mexican Hat Wavelet filters –Multi-frequency Matrix filters –Matched Multi-filter code NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 23 WorkstationGridGain short330 m25 m13 long15342 m955 m16

24. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Fusion Application porting –9 applications have been ported to give relevant scientific results Tooling: –Data mgt. tool development with goal of multi-machine analysis –GIF Portal for launching Generic Algorithm-based applications –Use of Kepler workflow engine (bridge to EUFORIA project) NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 24

25. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Fusion Developments ISDEP MC code follows ion trajectories inside plasma –Self-consistent plasma profiles: intro. of non-linear effects –Divertor Studies: Map of 3D fluxes on wall of device –Tokamak geometry –Ion heating ASTRA-MaRaTra –First complex fusion workflow between applications running on different platforms –ASTRA: SGI Application –MaRaTra: Grid Infrastructures NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 25 ASTRA MaRaTra

26. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Activity Coordination Activity Management –All milestones and deliverables have been achieved –Maintain the RESPECT program –Encouraged community interaction via the User Forum –Encourage participation in meetings through “travel money”:  Financed 6 people to attend EGEE’08 and UF4  Sponsor of the GMAC’09 workshop –Contributed to EGEE  EGI migration via SSC Workshops –Collaboration with MathWorks for MATLAB on the grid Regional Coordination –Design, implementation, and filling of Application Database –First line support, document review, etc. –Liaison activities increasingly important as EGI approaches NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 26

27. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 RESPECT Identify third-party software that works well with gLite. –http://technical.eu-egee.org/index.php?id=290http://technical.eu-egee.org/index.php?id=290 Simplified Access –P-GRADE, Ganga, Migrating Desktop, g-Eclipse, i2glogin, Virtual Control Room Workload Management –GridWay Metascheduler, DIANE New Resources –GRelC, Instrument Element Infrastructure Services –StoRM NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 27

28. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 User Forum NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 28 UF1 (CERN) UF2-OGF20 (Manchester) UF3 (Clermont-Ferrand) UF4-OGF25 (Catania) http://technical.eu-egee.org/index.php?id=148

29. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Issues Technical Issues –Fragility of applications with upgrades –Ease of use (availability of Java APIs) –SAM  Nagios transition for VO-specific tests –Firewall configurations and data transfers –MPI support Administrative Issues –Late recruiting –Unresponsive partners Systemic Problems –Visibility of the NA4 support services. –Underutilization of those support services. NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 29 Followed up in project Largely resolved Emphasis in Year 2

30. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Deviations from Work Plan NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 30 Task Consumed Effort (PM8) Planned Effort (PM8)Deviation (%) VO Support13.715.1-9.7% App. Port. Support39.460.3-34.6% Dir. User Support36.649.6-26.2% High Energy Phys.54.348.5+11.8% Life Science40.732.9+23.9% Earth Science20.120.4-1.1% Grid Observatory21.811.0+99.2% Comp. Chemistry26.820.431.7% Astron. & Astro.17.617.22.3% Fusion20.717.220.1% Activity Mgt.24.020.9+14.9% Reg. Coord.49.457.4-14.0% Cross-Activity Tasks102.9 0.0% TOTAL468.0473.7-1.2% Higher spending than planned. Expect rate to continue. Becomes additional unfunded contributions. Significant fraction of expended effort. Lower spending than planned. Slow start up. Low visibility and under- utilization of services.

31. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Plans for Year 2 Support Activities: –Improve visibility and use of all support services –Publicize the seed resources for new users and new VOs –Work on transition to EGI support structures Strategic Discipline Clusters –Continue current scientific activities –Work on transition to the EGI SSC models Management –Continue coordination activities –Make more use of community building funds –Enhance cooperation with NA2 to increase dissemination NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 31

32. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Specialized Support Centers No major structural changes: –NA4 Steering Committee  User Forum Steering Committee –Strategic Discipline Clusters  Specialized Support Centers Each SSC: –Must be much more autonomous than now –Must find and attract financial and political support –Must be the center of gravity for grid use within their communities It will be a hard challenge to have fully functional SSCs in time for the start of EGI. NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 32

33. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Summary Three principal tasks of NA4 have worked well. User Community –13000 users, 220 applications, 112 registered VOs –Majority of use from 23 core VOs –Overall CPU use increased by factor of 2 Scientific impact –Shown results could only be achieved with the grid. –User Forum 4 program and Book of Abstracts –Detailed achievements provided in DNA4.4.1 Future Work –Improve visibility and utilization of support services –Guide formation of SSCs for the EGI transition NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 33

34. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Seed Resources Limited pool of computing resources for: –Facilitating use of the infrastructure by new users and communities. –Stable resources with full set of services for porting new applications. Seed resources available since January 2009: –Sites: CYFRONET, AUTH, RAL, GRIF –Resources: CPU Cores = 275, Disk = 27 TB Used principally for porting although two VOs (ETICS and Climate-G) will take advantage of them. Need to increase visibility of the resources. NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 34

35. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Other Plots NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 35

36. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Other Plots NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 36

37. Enabling Grids for E-sciencE EGEE-III INFSO-RI-222667 Other Plots NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 37 Applications by Domain RegionApplications Asia_US0 Benelux7 Central Europe0 CERN/DE/CH6 FR/UK/IRL10 IT152 Northern Europe10 Russia0 SEE30 SWE6