1. Extending WeNMR e-Infrastructure outside Europe Marco Verlato INFN (National Institute of Nuclear Physics) Division of Padova Italy marco.verlato@pd.infn.it EGI Community Forum 2012, Munich, Germany, March 26 th – 30 th 2012

2. 4 Project Coordinator: Prof. Alexandre M.J.J. Bonvin, Utrecht University, NL Contract n°: RI-261572 Project type: CP-CSA Duration: 36 months (Oct.2013) Total budget: 2’434’000 € EC Funding: 2’150’000 € A Worldwide e-Infrastructure for NMR and structural biology Utrecht University, Bijvoet Center for Biomolecular Research, NL Johann Wolfgang Goethe Universität Frankfurt a.M., Center for Biomolecular Magnetic Resonance DE University of Florence, Magnetic Resonance Center, IT Istituto Nazionale di Fisica Nucleare, Padova, IT Raboud University, Nijmegen, NL University of Cambridge UK European Molecular Biology Laboratory, Hamburg, DE Spronk NMR Consultancy, LT + Academia Sinica,TW expected Spring 2012 The team The Project www.wenmr.eu 1 st VRC recognized by EGI Science Gateway  www.wenmr.eu

3. Operate and further develop a user-friendly e-Science gateway for the NMR and SAXS communities (link to ESFRI Instruct) Establish a virtual research platform for (interaction with) the user community Provide support to software developers, users and other e- Infrastructure projects Foster the adoption and use of e-Infrastructure in a wide range of flanking disciplines within the life sciences Operate and consolidate the eNMR Grid infrastructure and to extend it to interoperate with other worldwide Grid initiatives Develop a model to ensure sustainability of the project WeNMR main objectives

4. The VRC portal: www.wenmr.eu

5. Our policy is to shield as much as possible the end user from the Grid and all middleware related issues and commands For this, we chose to develop mainly web portal providing “protocolized” access to the Grid To facilitate operation, we use when possible robot certificates 19 portals are currently embedded in the WeNMR gateway and provide services for NMR and SAXS spectroscopists 10 of these run calculations on the Grid The WeNMR service portfolio

6. CPU intensive Includes parallel jobs (Gromacs portal) Limited data transfer (except for software deployment) Usually low I/O Usually no storage required Software characteristics Our main requirements: Lots of CPUs Queues of various lengths (including short ones) Our main requirements: Lots of CPUs Queues of various lengths (including short ones)

7. Why going worldwide? All potential users of the Grid! Currently only if registered with the enmr.eu VO ~ 2000 registered users (~125 with grid access) ~ 25000 served runs since June 2008 ~ 12% on the GRID Example: HADDOCK world map

8. Worldwide user distribution #395 VO registered users (March 21 st ) in 41 countries #95 users (24%) in 11 countries outside EU (#57 in USA) Steady growing rate ~ 10 users/month

9. WeNMR planned a grid interoperability activity to start in its second year Built on Network Activities aimed at involving worldwide grids and e-Science projects in its first year Goal: worldwide grid extension outside Europe Towards EMI/UMD based infrastructures  South Africa  gLite √  Latin America  gLite √  Asia-Pacific  gLite √ Towards non-EMI/UMD based infrastructures  OSG/SBGrid in USA  VDT (Globus, Condor, etc.) o WeNMR project milestone by October 2012  Asian grids o Garuda in India  Globus o CNGrid in China  GOS Worldwide grid extension

10. Collaborations (MoU) relevant for this activity already established with the following projects: ‒ CHAIN (Co-ordination and Harmonisation of Advanced e-Infrastructures) ‒ EUMEDGRID -Support (Sustainability of e-Infrastructures across the Mediterranean) ‒ GISELA (Grid Initiatives for e-Science virtual communities in Europe and Latin America) ‒ EU-IndiaGrid2 (Sustainable e-Infrastructures across Europe and India) ‒ EMI and EGI ‒ SBGrid (Software Consortium for Structural Biology) and OSG (Open Science Grid) in US Attended networking events since 2010: ‒ WeNMR at CHAIN KoM (Roma, 13-14 Dec.10) ‒ WeNMR at EU-IndiaGrid2 Workshop (Delhi, 13-16 Dec.10) ‒ WeNMR at OSG All Hands (Boston, 7-11 Mar.11) ‒ WeNMR at ISGC 2011 / OGF31 (Taipei, 21-25 Mar.11) ‒ WeNMR at GISELA & CHAIN Workshop (Vilnius in April, Lyon in Sep.11 ) ‒ WeNMR at SC11 Exposition (Seattle, 14-17 Nov.11) ‒ WeNMR at ISGC 2012 & CHAIN Workshop (Taipei, 26 Feb.-2 Mar.12) Building on Network Activities

11. Academia Sinica of Taiwan ready to join WeNMR Institute of Biological Chemistry Institute of Physics and ASGC Detailed program defined for contributing to all activities: Networking, dissemination and outreach  Translating documentation/pub. abstracts into Chinese  Covering major NMR, Biophysics and Struct. Biology events in AP area Services activities  Allocate and operate ASGC resources for WeNMR VO  Dedicated queues for MD simulation with multi-threaded GROMACS  Training material/FAQ/User support in Chinese Joint Research activities  Developing protocols for joint usage of SAXS and NMR with docking  Enhancing the working capacity of the SAXS beamline at the National Synchrotron Radiation Research Centre (NSRRC) in Taiwan New partner in Asia-Pacific

12. Worldwide resource distribution 40 central grid services in EU 0.3k 0.2k 2k 35k 27 resource centres in Europe, ~35k CPU-cores 5 resource centres outside Europe TW-eScience (2k CPU-cores), Taiwan UNLP+UFRJ+ULA (0.3k CPU-cores), GISELA UJ (0.2k CPU-cores), SAGrid Countries with users Countries with users & resources

13. Keeping aligned with EGI monitoring/accounting tools: Operation of a dedicated instance of Gstat 2.0 Operation of a dedicated Nagios server Operation of a dedicated WMSMonitor server Operation of dedicated DGAS servers to complement APEL Collaboration with EGI-InSPIRE NA3 and EGI.eu UCST teams for integration with the EGI VO Dashboards Grid operations

14. Resource usage Taiwan and GISELA resources usage comparable to EU sites of the same size

15. Resource usage (last six months) ~1 Million Normalized CPU-hours (kSI2K)/month on average it was ~0.6M in the previous 6 months (+55%) 77% by Dutch NGI, 2% for outside EU sites ~170k jobs/month on average it was ~140k in the previous 6 months (+21%) 54% by Dutch NGI, 7% from outside EU sites 38%50% 8% enmr.eu biomed lsgrid enmr.eu biomed

16. Extension to US Open Science Grid OSG is already inter-operating with EGI (e.g. in HEP) http://sbgrid.org Collaboration with SBGrid Consortium well established WeNMR access provided as option to users registering with SBGrid Science Portal since one year 57 US users (DOEGrids CA) currently registered with enmr.eu VO (14%) SBGrid is also a VO of OSG supported by 26 OSG sites, >60k CPU-cores gLite clients and application software path available on WNs

17. Interoperability effort started in March 2011 Discussed at OSG AHM and at EGI-UF last year with OSG, EGI and SBGrid directors Joint team with OSG-SBGrid-WeNMR-EGI personnel set up Use of SBGrid VO to access OSG/SBGrid resources Two options for submitting WeNMR jobs to OSG: using gLite UI / WMS / SE / LFC enabled with SBGrid VO using the SBGrid glideinWMS submission system GlideinWMS is the OSG recommended pilot-based job submission system WeNMR-SBGrid Interoperability Plan

18. Using gLite-WMS Existing gLite UI and WMS enabled with SBGrid VO, new LFC server set up for SBGrid VO at INFN-Padova centre Data transfer from/to 10 OSG SEs with registration on LFC successful using lcg-utils from both gLite UI and OSG WNs CS-Rosetta software installed from remote in 14 OSG CEs CS-Rosetta test jobs successful submitted through gLite- WMS (with minimal changes of bash scripts) and run on OSG WNs

19. Using SBGrid glideinWMS Condor submit node installed on a gLite UI at INFN Configured to work with SBGrid VO Frontend at HMS WeNMR sw installed from remote on $OSG_APP area via condor job or globus-job-run on the CE head node CS-Rosetta test jobs successfully submitted in two ways: direct condor_submit (JDL  condor submit file translation needed) via SAGA Python API implemented by LSU (www.saga-project.org) Glideins are submitted by the OSG Factory with the SBGrid VO proxy WeNMR jobs can be submitted with the enmr.eu VO proxy allows interaction with WeNMR existing LFC and SEs services allows jobs to be accounted in Gratia system under enmr.eu VO (and its VO groups/roles)

20. Using SBGrid glideinWMS/Architecture

21. Using SBGrid glideinWMS/Accounting Job statistics provided by: Condor view Gratia

22. Next steps Need of a SAGA implementation with both Condor and gLite/EMI-WMS adaptors SAGA-LSU has a Condor adaptor but not the glite-WMS one JSAGA has the gLite-WMS adaptor but not the Condor one Testing massive job submission with the devel. portal Put the stuff in production starting from the most demanding applications (Amber, CS- Rosetta, Haddock, …) or the most popular among US users same robot proxy and same SAGA based script behind the portal for handling job submission to both gLite-WMS and Condor

23. Extension to India and China Grids Leverages on the collaboration with the CHAIN project CHAIN has proposed an interoperability plan UMD, GARUDA (India), GOS (China) and OurGrid (Brazil) middlewares considered Application-level vs Service-level interoperability Based on OGF standards: OGSA-BES, JSDL (job execution) SAGA, DRMAA (client API) GLUE (info system) SAML (security) GridFTP, SRM (data management) A possible solution for WeNMR

24. Interoperability with GARUDA-India DRMAA API gLite UI WMS SAGA layer GARUDA-CE (GT4 based) GridWay (Globus) 15 sites, 6200CPUs (70TF), 0.22PB Leveraging also on the collaboration with EU-IndiaGrid 1&2 projects Middleware based on GLOBUS 4.x and the meta scheduler GridWay 5.7 DRMAAv1 implementation packaged with GridWay Interoperability approach through a SAGA DRMAA adaptor is under investigation

25. Interoperability with CNGrid-GOS SAGA GOS adaptor gLite UI WMS SAGA layer GOS Node GOS Services 11 sites, >430TFlops, 2.2PB, >1400 users Middleware strongly based on GLOBUS, but actually completely rewritten in Java with a quite different layout This stage of the project is closing, they are considering to evolve in new directions Positive feedback to CHAIN plan, effort allocated for developing adaptors and service level interfaces

26. Summary WeNMR grid extended towards gLite/EMI based infrastructures in South Africa, Latin America and Asia- Pacific area. More sites to come (e.g. from Mediterranean) Very successful collaboration with SBGrid/OSG allowed to achieve interoperability well before the project milestone More long term, good perspectives for extending towards Indian and China grids, thanks to collaboration with CHAIN Sustainability of grid infrastructures means nothing without an active user base and demonstrated usage

27. Acknowledgments The WeNMR/EGI/SBGrid/OSG team Alexandre M.J.J. Bonvin Christophe Schmitz Marco Verlato Eric Frizziero Nuno Loureiro-Ferreira Piotrek Slitz Jan Stoke-Rees Peter Doherty Dan O’Donovan Ruth Pordes Gabriele Garzoglio Igor Sfiligoi Marco Mambelli Chander Sehgal Burt Holzman Douglas Strain