EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks C. Loomis (CNRS/LAL) NA4 Activity Manager EGEE-III First Review, June, 2009 NA4: User Community Support and Expansion

Enabling Grids for E-sciencE EGEE-III INFSO-RI Activity Overview NA4 - C. Loomis - EGEE-III First Review June CountryPMFTE Austria60.3 Belgium120.5 CERN Cyprus120.5 Czech Republic France Germany472.0 Greece923.8 Hungary Israel120.5 Italy Netherlands291.2 Norway301.3 Poland421.8 Russia662.8 Slovakia241.0 Spain Sweden160.7 UK411.7 TOTAL Partners 287People 19Countries NA4: 19%

Enabling Grids for E-sciencE EGEE-III INFSO-RI Tasks TNA4.1: Support –Virtual Organization Support (VOS) –Application Porting Support (APS) –Direct User Support (DUS) TNA4.2: Strategic Discipline Clusters –High Energy Physics (HEP) –Life Sciences (LS) –Earth Sciences (ES) –Grid Observatory (GO, CS) –Computational Chemistry (CC) –Astronomy & Astrophysics (AA) –Fusion (F) TNA4.3: Activity Coordination –Activity Management –Regional Coordination NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI User Community Organization NA4 - C. Loomis - EGEE-III First Review June User VO Domain User Community Grid Auth. Clusters

Enabling Grids for E-sciencE EGEE-III INFSO-RI Community & Use NA4 - C. Loomis - EGEE-III First Review June DomainVOsUsers AA20373 CC4347 CS421 ES7142 F268 HEP LS9379 MV OTH TOTAL Around Registered Users Consistent doubling every months. EGEE-III = Y2EGEE-II = Y1 Accounting Portal: CIC Portal:

Enabling Grids for E-sciencE EGEE-III INFSO-RI CPU Utilization by Domain Domain VOs (>0%) VOs (>10%) VO Names (>10%) AA113astro.vo.eu-egee.org, auger, virgo CC42compchem, trgrida CS21imath.cesga.es ES41esr F11fusion HEP324alice, atlas, cms, lhcb LS71biomed MV125aegis, balticgrid, see, seegrid, vo.gear.cern.ch OTH192geant4, theophys UNK793bg, litgrid, vo.nanocmos.ac.uk NA4 - C. Loomis - EGEE-III First Review June Registered VOs 171 “Visible” VOs 23 “Core” VOs 4167 “Core” Users DomainY1Y2Y2/Y1 AA CC CS ES F HEP LS MV OTH UNK TOTAL x increase overall HEP largest users / contributors AA/ES/OTH show strong increase CPU Use: 1K-SI2K-Month

Enabling Grids for E-sciencE EGEE-III INFSO-RI Applications NA4 - C. Loomis - EGEE-III First Review June Alt. link:

Enabling Grids for E-sciencE EGEE-III INFSO-RI Virtual Organization Support VO Management Developments –Improving the VO registration information –Integration of collaborative tools with VO information –Expansion of SAM testing framework for non-LHC VOs Documentation and Support Provision –Links to VO documentation: –Liaison between operations and VO managers VO Tools Identification –Identified problems with VOMS functionality –Worked in collaboration with JSPG on changes to policies related to VO management NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Application Porting Support Consultancy and Porting –15 applications ported; ~10 applications being ported Training –Group collects, prepares, reorganizes training materials and offers those as customized training packages for users –Direct participation in NA3 training events Provision of Infrastructure Services –Group leader is VO manager for the NA4 –Partly responsible for Application Database Public Relations –Writing of success stories of ported applications to increase visibility and to help others with similar applications –“EGEE App. Porting Support Group” won Best Demo prize at EGEE’08 NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Application Porting Support NA4 - C. Loomis - EGEE-III First Review June applications being ported 15 applications ported

Enabling Grids for E-sciencE EGEE-III INFSO-RI Direct User Support Ticket Handling –DUS support unit part of GGUS since mid-September –Have taken 2-person, 2-week shifts to treat tickets –The number of tickets assigned to DUS has been small Documentation and Use Cases –Reviewed and accessed existing documentation –Writing new documentation to fill identified gaps –Working with clusters and other teams to improve their documentation NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Ganga/DIANE, AMGA, Dashboard –Used by 1000s in HEP, strong adoption by other communities –Ganga/DIANE tutorials: NSS IEEE, Helsinki, BalticGrid –Dashboard tutorial: UF4/OGF25 –CERN Training for Trainers Grid validation for LHC data taking: CCRC’08, STEP’09 –4 expts., 3 grid infra., 100s of sites, O(PB) data –Sustained 4GB/s CERN  T1, O(100K) jobs/day High Energy Physics NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Technology Transfer Collab. HEP and Fusion clusters (since EGEE’08) –Porting of specific fusion applications using Ganga/DIANE –Results of the collaboration shown during UF4/OGF25 Lattice QCD (in production since 2007) –Running autonomously on a daily basis using Ganga/DIANE –Sustained rate of 1000 concurrent jobs, 750 CPUs and more than 20TB of data transferred GEANT4 simulation toolkit (since 2005) –Widely used by astroparticle physics, medical applications, radiation studies, as well as HEP –Validation of the new releases performed regularly on EGEE grid, again using Ganga/DIANE NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Life Sciences Major calculations: –WISDOM ( –System biology on cancer data –Genetic linkage analysis for disease loci –Identification of causes for coronary artery diseases Tooling Support –AMGA –Medical Data Manager –MOTEUR –Taverna2 Plug-In NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Nature Genetics Article Genome-wide haplotype analyses of complex human diseases –Study the impact of DNA mutations on human coronary diseases –Very CPU intensive analysis to study the impact of correlated (double, triple) DNA mutations EGEE grid deployment –1926 Coronary Artery Disease patients; 2938 healthy controls –378,000 Single Nucleon Polymorphisms = local DNA mutations –8.1 million combinations tested in less than 45 days (instead of >10 years on a single Pentium 4) Results in Nature Genetics Mar (D. Tregouet et al) –Major role of mutations on chromosome 6 was confirmed. NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Earth Sciences User and application support Dissemination –Session at European Geosciences Union (EGU) in 2008 –Special issue of journal with 12 peer-reviewed papers –2 PhDs and 5 papers based on Geocluster results from EGEE Specialized tools –Data distribution, file explorer, storage access systems, workflow tools, … NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Pesticide risk assessment and management in Europe –FP6 EU project –BRGM France + 14 partners in 9 countries Creation of a large database including 4 million scenarios (climate, soil, pesticides, …). Successful results with the first 2 million scenarios obtained with EGEE running around 4800 jobs/day. Exploitation of database and results by all partners. –Creation of one SME in France for agriculture consultancy. Footprint NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Grid Observatory Created the Grid Observatory Portal –Store and publish monitoring information for analysis Reaching out to CS community: –EGEE’08: Grid Community Meeting –UF4/OGF25: Joint session “From Grid Monitoring to Analysis” –Grid Meeting Autonomic Computing (GMAC’09) at ICAC NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Grid Observatory Portal NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Computational Chemistry Analysis of grid licensing models. Expanding membership –Training of young researchers –Availability of necessary software packages Tooling: –Chempo, Charon, ECCE, Wien2K –Parallel version of GAMESS NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Computational Campaigns Chemical reactions –N + N 2, O + O 2 and F + HD –Thermal rate coefficients Nanotube modeling P-Grade port of ABC program NA4 - C. Loomis - EGEE-III First Review June Executor: executed as many times in parallel as many parameters are generated by “Generator” Collector: collects all output files into a single TAR file Generator: generates input files with different parameters (currently 4 input)

Enabling Grids for E-sciencE EGEE-III INFSO-RI Astronomy & Astrophysics Development of active community –Large number of applications ported to the grid –Focused training and dissemination Tooling: –Management of parameter sweep applications –Scheduling and bookkeeping systems –Visualization Interaction with EuroVO NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Planck Satellite Launched 14 May will be in L2 orbit in early July. INAF: Ported full LFI mission simulation to EGEE IFCA: Ported several codes to LFI Data Processing Center: –Mexican Hat Wavelet filters –Multi-frequency Matrix filters –Matched Multi-filter code NA4 - C. Loomis - EGEE-III First Review June WorkstationGridGain short330 m25 m13 long15342 m955 m16

Enabling Grids for E-sciencE EGEE-III INFSO-RI Fusion Application porting –9 applications have been ported to give relevant scientific results Tooling: –Data mgt. tool development with goal of multi-machine analysis –GIF Portal for launching Generic Algorithm-based applications –Use of Kepler workflow engine (bridge to EUFORIA project) NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Fusion Developments ISDEP MC code follows ion trajectories inside plasma –Self-consistent plasma profiles: intro. of non-linear effects –Divertor Studies: Map of 3D fluxes on wall of device –Tokamak geometry –Ion heating ASTRA-MaRaTra –First complex fusion workflow between applications running on different platforms –ASTRA: SGI Application –MaRaTra: Grid Infrastructures NA4 - C. Loomis - EGEE-III First Review June ASTRA MaRaTra

Enabling Grids for E-sciencE EGEE-III INFSO-RI Activity Coordination Activity Management –All milestones and deliverables have been achieved –Maintain the RESPECT program –Encouraged community interaction via the User Forum –Encourage participation in meetings through “travel money”:  Financed 6 people to attend EGEE’08 and UF4  Sponsor of the GMAC’09 workshop –Contributed to EGEE  EGI migration via SSC Workshops –Collaboration with MathWorks for MATLAB on the grid Regional Coordination –Design, implementation, and filling of Application Database –First line support, document review, etc. –Liaison activities increasingly important as EGI approaches NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI RESPECT Identify third-party software that works well with gLite. – Simplified Access –P-GRADE, Ganga, Migrating Desktop, g-Eclipse, i2glogin, Virtual Control Room Workload Management –GridWay Metascheduler, DIANE New Resources –GRelC, Instrument Element Infrastructure Services –StoRM NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI User Forum NA4 - C. Loomis - EGEE-III First Review June UF1 (CERN) UF2-OGF20 (Manchester) UF3 (Clermont-Ferrand) UF4-OGF25 (Catania)

Enabling Grids for E-sciencE EGEE-III INFSO-RI Issues Technical Issues –Fragility of applications with upgrades –Ease of use (availability of Java APIs) –SAM  Nagios transition for VO-specific tests –Firewall configurations and data transfers –MPI support Administrative Issues –Late recruiting –Unresponsive partners Systemic Problems –Visibility of the NA4 support services. –Underutilization of those support services. NA4 - C. Loomis - EGEE-III First Review June Followed up in project Largely resolved Emphasis in Year 2

Enabling Grids for E-sciencE EGEE-III INFSO-RI Deviations from Work Plan NA4 - C. Loomis - EGEE-III First Review June Task Consumed Effort (PM8) Planned Effort (PM8)Deviation (%) VO Support % App. Port. Support % Dir. User Support % High Energy Phys % Life Science % Earth Science % Grid Observatory % Comp. Chemistry % Astron. & Astro % Fusion % Activity Mgt % Reg. Coord % Cross-Activity Tasks % TOTAL % Higher spending than planned. Expect rate to continue. Becomes additional unfunded contributions. Significant fraction of expended effort. Lower spending than planned. Slow start up. Low visibility and under- utilization of services.

Enabling Grids for E-sciencE EGEE-III INFSO-RI Plans for Year 2 Support Activities: –Improve visibility and use of all support services –Publicize the seed resources for new users and new VOs –Work on transition to EGI support structures Strategic Discipline Clusters –Continue current scientific activities –Work on transition to the EGI SSC models Management –Continue coordination activities –Make more use of community building funds –Enhance cooperation with NA2 to increase dissemination NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Specialized Support Centers No major structural changes: –NA4 Steering Committee  User Forum Steering Committee –Strategic Discipline Clusters  Specialized Support Centers Each SSC: –Must be much more autonomous than now –Must find and attract financial and political support –Must be the center of gravity for grid use within their communities It will be a hard challenge to have fully functional SSCs in time for the start of EGI. NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Summary Three principal tasks of NA4 have worked well. User Community –13000 users, 220 applications, 112 registered VOs –Majority of use from 23 core VOs –Overall CPU use increased by factor of 2 Scientific impact –Shown results could only be achieved with the grid. –User Forum 4 program and Book of Abstracts –Detailed achievements provided in DNA4.4.1 Future Work –Improve visibility and utilization of support services –Guide formation of SSCs for the EGI transition NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Seed Resources Limited pool of computing resources for: –Facilitating use of the infrastructure by new users and communities. –Stable resources with full set of services for porting new applications. Seed resources available since January 2009: –Sites: CYFRONET, AUTH, RAL, GRIF –Resources: CPU Cores = 275, Disk = 27 TB Used principally for porting although two VOs (ETICS and Climate-G) will take advantage of them. Need to increase visibility of the resources. NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Other Plots NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Other Plots NA4 - C. Loomis - EGEE-III First Review June

Enabling Grids for E-sciencE EGEE-III INFSO-RI Other Plots NA4 - C. Loomis - EGEE-III First Review June Applications by Domain RegionApplications Asia_US0 Benelux7 Central Europe0 CERN/DE/CH6 FR/UK/IRL10 IT152 Northern Europe10 Russia0 SEE30 SWE6