EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Towards Seamless Grid Computing The EGEE Experience on Interoperable Grid Infrastructures Erwin Laure EGEE-II Technical Director

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April eScience Science is becoming increasingly digital, needs to deal with increasing amounts of data and computational needs Simulations get ever more detailed –Nanotechnology – design of new materials from the molecular scale –Modelling and predicting complex systems (weather forecasting, river floods, earthquake) –Decoding the human genome Experimental Science uses ever more sophisticated sensors to make precise measurements  Need high statistics  Huge amounts of data  Serves user communities around the world

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Scientific trends Scientific advances are more and more based on simulations using “virtual laboratories” Ulf Dahlsten, Former Director Emerging Technologies and Infrastructures, EU, predicts that “in five years 80 percent of all scientific papers in all areas will be made in virtual laboratories. Fifty percent of social science documents will go the same way in five to ten years.” The size of data an organization owns, manages, and depends on is dramatically increasing: –Ownership cost of storage capacity goes down –Data generated and consumed goes up –Network capacity goes up –Distributed computing technology matures and is more widely adopted

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April EGEE Main Objectives Operate a large-scale, production quality grid infrastructure for e-Science Attract new resources and users from industry as well as sciences – Flagship grid infrastructure project co-funded by the European Commission – Now in 2 nd phase with 91 partners in 32 countries

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April EGEE – What do we deliver? Infrastructure operation –Sites distributed across many countries  Large quantity of CPUs and storage  Continuous monitoring of grid services & automated site configuration/management  Support multiple Virtual Organisations from diverse research disciplines Middleware –Production quality middleware distributed under business friendly open source licence  Implements a service-oriented architecture that virtualises resources  Adheres to recommendations on web service inter- operability and evolving towards emerging standards User Support - Managed process from first contact through to production usage –Training –Expertise in grid-enabling applications –Online helpdesk –Networking events (User Forum, Conferences etc.)

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April sites 48 countries 50,000 CPUs 13 PetaBytes >5000 users >200 VOs >140,000 jobs/day Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences … 32%

Enabling Grids for E-sciencE EGEE-II INFSO-RI Users and resources distribution EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April EGEE Grid Management Structure Operations Coordination Centre (OCC) –management, oversight of all operational and support activities Regional Operations Centres (ROC) –providing the core of the support infrastructure, each supporting a number of resource centres within its region –Grid Operator on Duty Resource centres –providing resources (computing, storage, network, etc.); Grid User Support (GGUS) –At FZK, coordination and management of user support, single point of contact for users

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Example: GridMap Monitoring Visualization 9

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Registered Collaborating Projects Applications improved services for academia, industry and the public Support Actions key complementary functions Infrastructures geographical or thematic coverage 25 projects have registered as of September 2007: web pageweb page

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April EGEE working with related infrastructure projects GINGIN

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April 2008 LHC Use of Multiple Grid Infrastructures 12

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April 2008 Why is Interoperability difficult? Grid infrastructures use different technologies –And even if same technologies are used they are usually heavily customized Only a few widely adopted standards –gridFTP, X.509 (but used differently!) –Prototypes: BES, JSDL, …  Production Grids are difficult to change – adopting standards takes time  Standards need to be stable before adoption Apart from technological differences, access policies also differ –Dialog among major Grid infrastructure providers started at last OGF22. Strong interactions between infrastructures and application community needed –HEP was driving interop efforts for LHC –Other applications can build on these experiences 13

Enabling Grids for E-sciencE EGEE-II INFSO-RI Access to Compute Resources PBS/Torque LSF Condor Load Leveler Sun Grid Engine ARC CREAM NAREGI Unicore OSG Nordugrid Naregi DEISA EGEE Teragrid GRAM v2/v4 GRAM v4WS EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI How to Start Understanding the differences –Compatibility matrix Domains that have to be linked for interoperability –Security –Information Services –Job Management –Data Management For interoperation you have to add –Monitoring –Accounting –Operational links and joint policies –Trouble ticket systems –Operational security EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Interoperability Matrix 1.Understand both middleware stacks 2.Identify the “common” interfaces 3.Create an interoperability matrix SRM Storage Control Protocol GSI/VOMS GridFTP GLUE v1 LDAP/GIIS GRAM OSG GSI/VOMS Security GridFTP Storage Transfer Protocol GLUE v1.2ARCSchema LDAP/BDIILDAP/GIISService Discovery GRAMGridFTPJob Submission EGEEARC EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Different Strategy Long term solution –Common interfaces –Standards Medium term solutions –Gateways –Adaptors and Translators Short term solutions –Parallel Infrastructures  User driven  Site driven EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Parallel Infrastructures User Driven –The user joins both grids  Uses different clients Depending on which interface –More work for the User  Required for each infrastructure –Keyhole approach  Restricts functionality –Method initially used by ATLAS  Split workload between grids EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Parallel Infrastructures Site Driven –The site joins both grids  Deploys both interfaces –User only sees their grid interface –More work for the site  Can only be supported by large sites Reduced resources –Use By FZK  Participating in EGEE, Nordugrid and D-Grid EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Gateway A gateway is a bridge between grid infrastructures –Single point of failure –Gateway breaks, grid disappears –Scalability bottleneck –All the load through one service Useful as a proof concept and to demonstrate the need NAREGI approach using glite-CE Gateway EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI Adaptors and Translators Adaptors allow connection Translators understand/modify information They are built into the middleware –The middleware can then work with both interfaces  Useful feature even when using standards! Requires modification to the grid middleware –Existing service interfaces can still be used Using in the GIN information System; most portals API Plugin EuroVO Workshop - April 2008

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Worldwide Grids APAC DEISA EGEE Naregi NDGF NGS OSG Pragma Teragrid GINGIN

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April How mature are we? Gartner Group Grid on the Computing in High Energy Physics conferences timeline Padova 2000 Beijing 2001 San Diego 2003 Interlaken 2004 Mumbai 2006 Victoria 2007 Slide courtesy of Les Robertson, LCG Project Leader

Enabling Grids for E-sciencE EGEE-II INFSO-RI From e-Infrastructures to Knowledge Infrastructures Network infrastructure connects computing and data resources and allows their seamless usage via Grid infrastructures Federated resources and new technologies enable new application fields: –Distributed digital libraries –Distributed data mining –Digital preservation of cultural heritage –Data curation → Knowledge Infrastructure NETWORK. INFRASTRUCTURE GRID. –INFRASTRUCTURE KNOWLEDGE. INFRASTRUCTURE 24 EuroVO Workshop - April 2008

25 ICT for Science: e-Infrastructures Linking at the speed of the light Sharing computers, instruments and applications Sharing and federating scientific data Astrophysics community Weather Forecast community Biomedics community Connecting the finest minds Sharing and federating the best scientific resources Building global virtual communities Mario Campolargo Acting Director Emerging Technologies and Infrastructures, EU European Information Space: Infrastructures, Services and Applications Workshop, Rome, October 2007

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Evolution European e-Infrastructure Testbeds Utility Service Routine Usage National Global

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April European Grid Initiative Need to prepare permanent, common Grid infrastructure Ensure the long-term sustainability of the European e-Infrastructure independent of short project funding cycles Coordinate the integration and interaction between National Grid Infrastructures (NGIs) Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines

Enabling Grids for E-sciencE EGEE-II INFSO-RI EuroVO Workshop - April Summary EGEE provides a dependable production quality Grid infrastructure to a wide variety of scientific disciplines. Collaborations on technical and political topics are key to implement a truly world-wide infrastructure Need to cover full spectrum: from individual sites, small scale Grids to world-wide infrastructures Grids are increasingly becoming an essential part of the scientific computing infrastructure – sustainability needs to be ensured