1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI service offerings to the CMMST community Gergely Sipos EGI.eu Technical Outreach Manager gergely.sipos@egi.eu European Grid Infrastructure ‘Towards a Chemistry, Molecular & Materials Science and Technology VRC’ Virtual Team project https://wiki.egi.eu/wiki/Towards_a_CMMST_VRC 16/May/2013

2. www.egi.eu EGI-InSPIRE RI-261323 Context of this talk https://wiki.egi.eu/wiki/Towards_a_CMMST_VRC Towards a CMMST VRC Virtual Team project –Virtual Research Communities (VRCs) are groups of like-minded individuals organised in EGI by discipline or computational model –VRC partnership is formalised with an MoU with EGI Resources consumed/needed from the National Grid Initiatives Joint activities (development, events, support,...) Communication channels –Project goal: Define the 1.structure of the CMMST VRC (incl. Organisation and funding model); 2.technologies, resources and services that already exist within EGI and could be used to satisfy the requirements of the CMMST VRC; 3.technologies that need to be developed or brought into EGI for the VRC; –Outcome: a proposal to establish a CMMST VRC in EGI Expected outcome today: Better understanding of 2. & 3. 2

3. www.egi.eu EGI-InSPIRE RI-261323 Outline EGI overview –Stakeholders, users, strategic activities EGI services for the CMMST community –User support structure –Ready-to-use scientific applications –Virtual Organisations / Grid sites –Federated cloud infrastructure –Community management and collaboration services –Tools: Workflows; Workflow integration; scientific grateway frameworks New initiatives –HPC-HTC integration 3

4. www.egi.eu EGI-InSPIRE RI-261323 European Grid Infrastructure European –Over 35 countries Grid –Secure sharing of IT resources Infrastructure –Computers (clusters) –Data –Applications –…. and beyond!! 4

5. www.egi.eu EGI-InSPIRE RI-261323 Key resource providers in EGI: National Grid Infrastructures 5 Metric Value (March 2013) Sites315 Nb. of CPU cores372,612 Disk (PB)180 PB Tape (PB)167 PB

6. www.egi.eu EGI-InSPIRE RI-261323 6 A multi-disciplinary e-infrastructure http://www.egi.eu/community/ User VO Virtual Research Community Members Virtual Organisations Research communities Most active VO groups: High Energy Physics (wLCG) Astronomy & Astrophysics Life Sciences (LSGC) Earth Sciences Computational Chemistry Fusion … ~22.000 users, ~230 Registered VOs: http://operations-portal.egi.eu/vo Sites Current VRCs: wLCG - High Energy Physics WeNMR – Structural biology LSGC – Life sciences (HMRC – Hydro-Meteorology) (CLARIN – Humanities) Sites VO GridCloud

7. www.egi.eu EGI-InSPIRE RI-261323 EGI, EGI.eu, EGI-InSPIRE EGI – A collaboration. An ecosystem. –Operates the infrastructure – and does other things too! (see later) –Aim: Support the digital European Research Area through a pan-European research infrastructure based on services federed from the NGIs EGI.eu – A Dutch fundation owned by the NGIs –To coordinate the work of EGI (operations, technology, user support, policy, community & communications, administration) –Sustainable, small organisation EGI-InSPIRE – An FP7 project –Supports EGI.eu and EGI to transition to sustainable operational model –May 2010 – April 2014 7

8. www.egi.eu EGI-InSPIRE RI-261323 What is an NGI? http://www.egi.eu/about/ngis/ The country's single point of contact for government, research communities and resource centres as regards ICT services for e- science. Manage the computing resources and support services they provide to the European Grid Infrastructure. Has named contact for: –International liaison –Support (User) –Operation (Infrastructure) 8

9. www.egi.eu EGI-InSPIRE RI-261323 EGI contributors National Grid Infrastructures (NGIs) Grid and cloud technologies: EMI, IGE, EDGI,... Science domain specific: wLCG, WeNMR, ScalaLife HMRC,... EGI-InSPIRE High level technologies: SCI-BUS, SHIWA, MAPPER,... EGI.eu Legal entities Projects Software + support (based on MoUs, SLAs) European Intergovernmental Research Organisations (e.g. CERN, EBI) Linking infrastructures: CHAIN EUDAT... 9

10. www.egi.eu EGI-InSPIRE RI-261323 EGI’s strategic focus areas Operational Infrastructure Virtual Research Environments Community & Coordination Operate a European wide grid infrastructure Build a federated cloud infrastructure Integration & operation of scientific environments Governance Communication (inreach, outreach) 10

11. www.egi.eu EGI-InSPIRE RI-261323 Use services Create services Operate services EGI services for research communities 5. Operation tools, dashboards 4. Collab. & communic. services 1. Scientific applications 11 Research infrastructure clusters 3. Application porting / hosting tools 2. Capacity (grid/cloud) Research collaborations Research infrastructures... Individual researchers

12. www.egi.eu EGI-InSPIRE RI-261323 Use services Create services Operate services 5. Operation tools, dashboards 4. Collab. & communic. services 1. Scientific applications 12 3. Application porting / hosting tools 2. Capacity (grid/cloud)... How is the CMMST community structured? Individual researchers Research collaborations Research infrastructures Research infrastructure clusters What does the CMMST community want to achieve in EGI? EGI services for research communities VT objective 1 VT objective 2 & 3

13. www.egi.eu EGI-InSPIRE RI-261323 1. Scientific applications Registered in EGI AppDB: http://appdb.egi.eu http://appdb.egi.eu –Browse by scientific discipline Access varies per application: –Public access –Via portal (username/pwd) –Register in VO (X.509 cert) –Download to client side –... Interact –Follow updates (RSS, Atom) –Rate & Comment –Contact provider/supporter By discipline 13

14. www.egi.eu EGI-InSPIRE RI-261323 2/1. Capacity – Grid access Join an existing Virtual Organisation: –http://operations-portal.egi.eu/vohttp://operations-portal.egi.eu/vo –‘Global’ Computational Chemistry Vos Compchem: https://www3.compchem.unipg.ithttps://www3.compchem.unipg.it Gaussian –National Computational Chemistry VOs In Turkey & Spain –Regional and multi-disciplinary Vos Many OR Setup a new Virtual Organisation: –http://operations- portal.egi.eu/vo/registrationWelcomehttp://operations- portal.egi.eu/vo/registrationWelcome –Request to support the VO is sent to site managers –Connect your own resources Get help from VO/NGI support teams 14 Grid sites at ~350 sites Grid middleware: UMD Unified Middleware Distribution (gLite, ARC, Unicore, dCache, SAGA, Desktop Grid) Domain applications and data

15. www.egi.eu EGI-InSPIRE RI-261323 EGI Grid use case example: batch processing Computing service Storage Service Site X of YOUR VO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 15 EGI Virtual Organisation job Your files

16. www.egi.eu EGI-InSPIRE RI-261323 Implementation: Auger VO Serving the Pierre Auger Collaboration Computing service Storage Service Site of AugerVO Information System Submit job query Retrieve output Create job definition Submit job (batch executable + <20 MB inputs) Broker service User environment publish state VO Management Service (VOMS) Login to your VO (With X509 cert) job Retrieve status & (small) output files Logging and bookkeeping service Job status Logging Read/write files 16 Auger Virtual Organisation on EGI job Cosmic ray files Copied from Colorado (USA), Malargüe (Arg) to grid storage Lyon (Fr) (CCIN2P3) Processing software is distributed to consortium members CPU intensive simulations done at various sites and results copied back to Lyon Central services for the VO are in Czech Republic (by CESNET)

17. www.egi.eu EGI-InSPIRE RI-261323 EGI Virtual Organisation EGI Grid use case example 2: data & meta-data management Computing service Storage service Site X of YOUR VO Information System Query User environment publish state VO Management Service (VOMS) Upload file Download file File Catalog and/or Metadata catalog Register file Lookup file File content Metadata 17 Login to your VO (With X509 cert) Your files

18. www.egi.eu EGI-InSPIRE RI-261323 Implementation: gLibrary http://glibrary.ct.infn.it 3TByte in total TIFF (100MB/A3 page) PDF (40-400MB/work) (1) Digitalise (2) Store (3) Catalogue + Meta-data (4) Browse De Roberto, an Italian writer of the XIX/XX century, born in Naples, but spending his life in Catania 18

19. www.egi.eu EGI-InSPIRE RI-261323 3. Application porting/hosting tools EGI Applications Database stores –Ready-to-use solutions that work with EGI –Reusable frameworks –Grouped in various categories (see below) Categories (numbers from February 2013) : –70 developer tool/framework/toolkit –46 science gateway & gateway framework –26 workflow system & workflow http://appdb.egi.eu 19 Domain specific services Science domain specific services Service access Off-the- shelf and custom VM images Data bases and repositories Scientific applications and algorithms Service orchestration Science gateways (Portals) Workflow systems Scientific algorithms, applications, workflows, databases To create custom environments for communities... GridCloud Very often only this part is unique Domain specific applications/services

20. www.egi.eu EGI-InSPIRE RI-261323 Workflow tools To define and run multi- step simulations Various workflow languages Various workflow environments Various workflow engines See workflows and workflow environments listed in EGI AppDB: http://appdb.egi.eu http://appdb.egi.eu Workflows 20

21. SHIWA Simulation Platform Benefits: Share workflows, re-use workflows of others Build ‘meta-workflows’ from smaller workflows – even if those use different languages & engines Combine multiple workflows and multiple grids and clouds Projects: SHIWA: Created the SHIWA Simulation Platform (2010-2012) ER-flow: Disemmination and support for the SHIWA technology (2012-2014). User Communities: Astro-physics – Istituto Nazionale di Astrofisica, Italy Comp. Chem. – Technische Universität Dresden, Ludwig-Maximilians-Universität München Heliophysics – Trinity College Dublin, University College London Life Science – Amsterdam Medical Centre + WE ARE LOOKING FOR NEW COMMUNITIES: http://www.erflow.eu/http://www.erflow.eu/ http://ssp.shiwa-workflow.eu/ Jobs & services 21

22. Generic use-cases DCI #1 DCI #2 CE SE CE SE DCI – distributed computing infrastructure; CE/SE – computing/storage element 22

23. Generic use-cases DCI #1 DCI #2 CE SE CE SE combine WFs find and re-use with own data on own CE? DCI – distributed computing infrastructure; CE/SE – computing/storage element 23

24. www.egi.eu EGI-InSPIRE RI-261323 Science gateways …are community-specific sets of tools, applications, and data collections that are integrated together via a web portal (or a desktop application) Main drivers: –Simple access –Modularity Gateways are domain/community specific, but the enabling technologies are typically not Science gateway frameworks: gUSE WS-PGRADE  MoSGrid, e-Bioinfra, DBrain, HellasGrid,...gUSE WS-PGRADE 21 in total - https://www.sci-bus.eu/science-gateways Catania Science Gateway Framework  CHAIN, GARR, AgINFRA,...Catania Science Gateway Framework 19 in total - https://gridp.garr.it/service-providers.html 24

25. www.egi.eu EGI-InSPIRE RI-261323 Science gateway Anatomy of a science gateway... App. 1 App. 2 App. N Embedded applications Administrator Power User Basic User Users with different roles and privileges Job submission engine Data transfer engine Grids, clouds, HPC, databases, etc. Authentiation & Authorization Authentiation & Authorization Local data (logs, authz, tmp, cache,...) 25

26. www.mosgrid.de Example: MosGrid portal https://mosgrid.de/portal 26 Job submission engine Data transfer engine HPC access with UNICORE Quantum Chemical Calculations based on approximated solutions of the Schrödinger equation Average scalability Gaussian, NWChem, Turbomole, … Molecular Dynamics based on forcefields describing molecular interactions Good scalability Gromacs, NWChem, Amber, … Docking Based on simplified forcefields Excellent scalability CADDSuite, FlexX, … Authentication

27. Example: CHAIN science gateway www.chain-project.eu/applications www.chain-project.eu/applications 27 1.Choose a science gateway 2.Sign in from an identity federation (user & psw) 3.Register in the gateway 4.Run pre-defined apps with your own data

28. Example: CHAIN science gateway www.chain-project.eu/applications www.chain-project.eu/applications 28 No run page for compchem apps  These can be used only from command line Request portal integration at http://applications.epikh.eu/ survey4sciencegateways http://applications.epikh.eu/ survey4sciencegateways

29. www.egi.eu EGI-InSPIRE RI-261323 Recent trends in science gateways More specialised gateways (each with a small nb. of users) Mixed grid and cloud use Transparent auth: –Robot certificates –Identity federations Mobile access Entry point on Facebook –Africa Grid Science Gateway (based on Catania SG Framework) 29

30. www.egi.eu EGI-InSPIRE RI-261323 2/2 Capacity – cloud access EGI Federated Cloud Local hypervisors Cloud resources at ~10 EGI sites OS Domain specific services in Virtual Machine Images OS EGI FedCloud interfaces 30 OCCI: VM Image management BDII: Information system X509: Authentication VOMS: Authorization APEL: Accounting OpenStack OpenNebula EmotiveCloud (Spain) Okeanos (OpenStack impl. in GR) WNoDeS (Italy) Any service, not only “batch processing” and “file based data management”

31. www.egi.eu EGI-InSPIRE RI-261323 Federated Cloud Task Force 1 st phase: Setup (Sep 2011 – Aug 2012) –Identify open source technologies (OpenStack, OpenNebula,...) and open standard (OCCI, CDMI, GLUE2,...) –Deploy a testbed 2 nd phase: Pilot use cases (Sep 2012 – Mar 2013) –Support early adopters –Collect feedback requirements –First user-facing services 3 rd phase: Towards production (Apr 2013 – Sep 2013) –Establish additional user facing services (e.g. Portals, tutorials, support) –‘Blueprint’ for resource providers who wish to join –Engage with additional use cases EGI Federated Cloud roadmap http://go.egi.eu/cloud 31

32. 32 IGI Portal – science gateway for the EGI Federated Cloud (portal.italiangrid.it) Domain specific service inside the VM

33. www.egi.eu EGI-InSPIRE RI-261323 4. Collaboration and communication services What we can provide for the VRC: Software services –Email list(s) –Single Sign On service: http://www.egi.eu/ssohttp://www.egi.eu/sso –Document repository: http://documents.egi.euhttp://documents.egi.eu –Training Marketplace: http://training.egi.euhttp://training.egi.eu –Applications Database: http://appdb.egi.euhttp://appdb.egi.eu –Requirements Tracker: http://go.egi.eu/requirementshttp://go.egi.eu/requirements Creation of content/materials –Case studies, news stories, newsletters, articles, prints, designs, blog posts, videos, leaflets, social network content, etc. Events –EGI annual meetings –F2f trainings and Webinars Outreach –EGI Champions: http://www.egi.eu/community/egi_champions/ (No Compchem champion yet! Delegate one?) http://www.egi.eu/community/egi_champions/ –International Science Grid This Week, HPCwire, HPCinthecloud and Datanami, etc. –Coordinated outreach (to ESFRIs) via the network of ‘NGI International Liaisons’ 33 To facilitate information exchange & knowledge integration in the community... In the pipeline: EGI Webinar Programme https://wiki.egi.eu/wiki/EGI_Webinar_Progr ammehttps://wiki.egi.eu/wiki/EGI_Webinar_Progr amme Catania Science Gateway Framework (15 May) AMGA Metradata catalogue International Workshop on Science Gateways: http://iwsg2013.org/, Zurich 3-5 Junehttp://iwsg2013.org/ EGI Technical Forum: http://tf2013.egi.eu/, Madrid 16-20 Septemberhttp://tf2013.egi.eu/

34. www.egi.eu EGI-InSPIRE RI-261323 Web Gadgets: Embed EGI software services into Websites The simplest way to reuse EGI collaboration services: embed EGI gadgets into your website Benefits –Customisability –Reusability –Compatibility Gadgets available for: –Applications Database –Training Marketplace –Requirement Tracker –Grid Relational DBs Catalogue (GRelC) Create your own gadget instance at: www.egi.eu/user-support/gadgets 34

35. www.egi.eu EGI-InSPIRE RI-261323 5. Operation Tools & dashboards EGI’s services: 1.Service registry (GOCDB) –Repository to maintain topology and other type of basic information about distributed services 2.Accounting infrastructure –To gather, process and visualise usage of distributed services 3.Service Availability Monitor (SAM) –To gather, process and visualise availability and reliability information about distributed services 4.Service discovery capability (GLUE and LDAP) –To store detailed and dynamic information about services. To discover services by features. 5.Helpdesk (GGUS, xGUS) –To troubleshoot problems or provide guidance about products and services. 35 Services to operate a multi-site, multi-national Research Infrastructure... Local NGIs Service Registry Accounting Monitoring Heldesk Research infrastructure Scientific services Infrastructure services Community portals Operation portals

36. www.egi.eu EGI-InSPIRE RI-261323 Example: Service Availability Monitoring (SAM) Framework for resource centres and service operators −Based on Nagios −Service-specific probes −Database to store service topology −Connected to a messaging network −Aggregate and browse results: −Operations Dashboard, MyEGI, etc. 36

37. www.egi.eu EGI-InSPIRE RI-261323 New project: HPC-HTC integration: https://wiki.egi.eu/wiki/EGI_EUDAT_PRACE_collaboration 37 Communities Mapper Virtual Physiological Human VERCE (Earthquake and seismology) Tools QCG-Icon File Transfer Service 3 Globus Online Protocols E-infrastructures EGIEUDATPRACE Middleware iRODS/GriffindCache SRM gsiftp/ftphttp/webdav UberFTPGlobus-url-copy EUDAT Data Staging QosCosGrid dCache (dcap)XROOT (rfio) XROOT DPM Objectives: –identify common data access and transfer tools and protocols –demonstrate real benefit for the involved communities –identify technology and/or organisational gaps and suggest measures for improvements Pilot 1: –Data sharing and uniform data access Pilot 2:. –Monitoring and resource discovery Conclusions expected –by mid August

38. www.egi.eu EGI-InSPIRE RI-261323 Conclusions EGI is based on a federated model –NGIs and projects are resource and service providers Established grid, emerging cloud infrastructure EGI is not just computing and storage –Human support network –Ready-to-use VOs and scientific applications –Application porting and hosting tools –Collaboration and communication services –Infrastructure operation tools and dashboards Various possibilities to use and extend EGI services in a CMMST VRC 38

39. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Thank you Questions? 39

40. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Backup slides 40

41. www.egi.eu EGI-InSPIRE RI-261323 EGI Infrastructure: The platform model Community Platform EGI resources (cores & storage) VM EGI Core Infrastructure Platform (Operation tools) EGI Cloud Infrastructure Platform (FedCloud) Virtual Research Communities EGI Collaboration platform wLCG Community Platform 41 Community Platform (UMD) Collaboration Platform (AppDB, Training Marketplace, DocDB,...)

42. www.egi.eu EGI-InSPIRE RI-261323 The current community platform: Unified Middleware Distribution (UMD) External technology providers: EMI (ARC, gLite, UNICORE, dCACHE), IGE (Globus), EDGI (desktop grids), SAGA, … UMD capabilities captured in UMD roadmap (last version 2012 May: https://documents.egi.eu/document/612)https://documents.egi.eu/document/612 Operations Provisioning Infrastructure Staged Rollout Staged Rollout Criteria Verification Criteria Verification Production Criteria Definition Criteria Definition External Technology Providers Deployed Software Helpdesk unit Deployed Software Helpdesk unit RequirementsSoftware Core infrastructure platform By EGI user communities and other stakeholders 42

43. www.egi.eu EGI-InSPIRE RI-261323 Structural biology – We-NMR project: Gromacs training environments Ecology – BioVel project: Remote hosting of OpenModeller service Linguistics – CLARIN project: Scalable ‘British National Corpus’ service (BNCWeb) Software development – SCI-BUS project: Simulated environments for portal testing Space science – ASTRA-GAIA project: Data integration with scalable workflows Some of the EGI FedCloud pilot use cases Join as pilot use case, or as provider of high level service on top of the EGI Cloud! http://go.egi.eu/cloud 43