1. EGI-InSPIRE EGI-InSPIRE RI-261323 www.egi.eu The European Grid Infrastructure Steven Newhouse Director, EGI.eu Project Director, EGI-InSPIRE 29/06/2016CoreGrid Workshop - August 2010 1

2. www.egi.eu EGI-InSPIRE RI-261323 The European Grid Infrastructure –Current Status and organisational structure An Evolving Infrastructure –Our challenges and possible directions Engagement with Computer Scientists –Opportunities and collaborations Wrap up Summary 29/06/2016CoreGrid Workshop - August 2010 2

3. www.egi.eu EGI-InSPIRE RI-261323 Infrastructure is the basic physical and organisational structures needed for the operation of a society or enterprise, or the services and facilities necessary for an economy to function 29/06/2016CoreGrid Workshop - August 2010 3 Infrastructure (Wikipedia) The Enterprise is the European Research Area

4. www.egi.eu EGI-InSPIRE RI-261323 A grid consists of distributed resources controlled by separate organisations that be systematically used securely by users external to that organisation Resources can include: –Commodity or HPC clusters –Disk or tape storage –Instruments –Data Archives or Digital Libraries What is a Grid? 29/06/2016CoreGrid Workshop - August 2010 4

5. www.egi.eu EGI-InSPIRE RI-261323 European Data Grid (EDG) –Explore concepts in a testbed Enabling Grid for E-sciencE (EGEE) –Moving from prototype to production European Grid Infrastructure (EGI) –Routine usage of a sustainable e-infrastructure 29/06/2016CoreGrid Workshop - August 2010 5 European Grid Infrastructure

6. www.egi.eu EGI-InSPIRE RI-261323 European Grid Infrastructure (Apr ‘10) 10000 users: +5% 243020 LCPUs (cores): +75% 40PB disk: +60% 61PB tape: +56% 15 million jobs/month: +10% 317 sites: +18% 52 countries: +8% 175 VOs: +8% 29 active VOs: +32% 6

7. www.egi.eu EGI-InSPIRE RI-261323 EGI Collaboration NGINGI NGINGI NGINGI NGINGI Research Community Research Community Research Community Research Community EIROEIRO EIROEIRO EGI.eu Research Community 29/06/2016CoreGrid Workshop - August 2010 7

8. www.egi.eu EGI-InSPIRE RI-261323 Coordination for European Grid resources –Established February 8th 2010 –Central policy & services needed to run a grid –Sustainable small coordinating organisation Governance & ownership by its participants –EGI Council votes linked to fees –Resources from within its participants Located in Amsterdam with approx. 40 staff –Coordinating core (~20 people) in Amsterdam 29/06/2016CoreGrid Workshop - August 2010 8 EGI.eu EGI and EGI.eu supported by EGI-InSPIRE project

9. www.egi.eu EGI-InSPIRE RI-261323 The EGI-InSPIRE Project 29/06/2016CoreGrid Workshop - August 2010 9 Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with €25M EC contribution –Project cost €72M –Total Effort ~€330M –Effort: 9261PMs Project Partners (50) EGI.eu, 39 NGIs, 2 EIROs Asia Pacific (8 partners) Funded Un-Funded

10. www.egi.eu EGI-InSPIRE RI-261323 Deploy Technology Innovation –Distributed Computing continues to evolve To include: Grids, Desktops, Virtualisation, Clouds, … Enable Software Innovation –Provide reliable persistent technology platform Tools built on gLite/UNICORE/ARC/Globus Support Research Innovation –Infrastructure for data driven research Support for international research (e.g. ESFRI) 29/06/2016CoreGrid Workshop - August 2010 10 EGI means Innovation

11. www.egi.eu EGI-InSPIRE RI-261323 Will come from outside EGI –Moving research technologies into production Partnership with technology projects –EMI (European Middleware Initiative) –IGE (Initiative for Globus in Europe) –EDGI (European Desktop Grid Initiative) –StratusLab –VenusC 29/06/2016CoreGrid Workshop - August 2010 11 Technology Innovation

12. www.egi.eu EGI-InSPIRE RI-261323 The DCI Projects ProjectEDGIEGI-InSPIREEMIIGEStratusLabVENUS-C Start Date 01/06/201001/05/2010 01/10/201001/06/2010 Duration (months) 2448363024 Total Budget 2,436,000 €72,000,000 €24,000,000 €3,693,000 €3,137,221 €8,803,046 € Funding from the EC 2,150,000 €25,000,000 €12,000,000 €2,350,000 €2,300,000 €4,500,000 € Total effort in person- month 28192412319277340639 12 29/06/2016CoreGrid Workshop - August 2010

13. www.egi.eu EGI-InSPIRE RI-261323 Venus-C EGI-InSPIRE Operational Infrastructure EDGI StratusLab DCI Projects - Taxonomy EMIIGE Resources Technology Users 13 29/06/2016CoreGrid Workshop - August 2010

14. www.egi.eu EGI-InSPIRE RI-261323 Advice to the EGI Community –The software base to build middleware e.g. OS, external software –The platform on which to build applications e.g. job submission interface, resource descriptions Components from within the EGI Community –Contained in the Unified Middleware Distribution –Defined integration and support through EGI –Describe what will be released and when. EGI Technology Roadmap 29/06/2016CoreGrid Workshop - August 2010 14

15. www.egi.eu EGI-InSPIRE RI-261323 EGI Technology Roadmap 29/06/2016 CoreGrid Workshop - August 2010 15 UMD Roadmap Components (Developed within EGI) EGI Technology Components Domain Specific Components Time  From outside EGI  Maintained by end-users

16. www.egi.eu EGI-InSPIRE RI-261323 Growing the use of the infrastructure –Diversify & expand the user communities –Provide expanded supported software services Deliver the resources more effectively –Add value in providing resources to researchers –Learn & extend lessons from other communities Empowering the end-users –Give more control to our ‘customers’ –Support for data intensive science Critical Influences on EGI 29/06/2016CoreGrid Workshop - August 2010 16

17. www.egi.eu EGI-InSPIRE RI-261323 17 Storage VMM Clusters VMM Clusters Staff Coordination by EGI.eu Technology assessment, Integrated Operations & User Support GEANT gLite ARC VMM: Virtual Machine Managers Staff Federated Virtualised Infrastructure 29/06/2016CoreGrid Workshop - August 2010

18. www.egi.eu EGI-InSPIRE RI-261323 Running Virtual Machine Images remotely –Site policy on trusting the image source –Sandboxing running VMs on a site –Provisioning infrastructure across a grid Resource Allocation and Provisioning –Mapping the VMs to the resources –Co-allocation of VMs, data and networking –Setting Service Level Agreements and monitoring Managing the provisioning infrastructure –Scalable WAN messaging system to integrate it –Autonomic monitoring of the resources Federated Virtualised Infrastructure 29/06/2016CoreGrid Workshop - August 2010 18

19. www.egi.eu EGI-InSPIRE RI-261323 Consider IP network providers –Supports traffic from different communities –Customised solutions within a generic framework –Standards drive integrated deployment And for sustainable e-Infrastructures? –Any application, any domain, any technology –A platform for domain specific innovation & use –Integration of any compliant resource 29/06/2016CoreGrid Workshop - August 2010 19 Be a Neutral Infrastructure

20. www.egi.eu EGI-InSPIRE RI-261323 Improve the efficiency of the infrastructure –The jobs, users & data will continue to increase –Effectiveness of the resources needs to match User input into upper middleware layers –VOs decide what services are deployed where –VOs manage their own deployed infrastructure –Empower the VOs to meet their own needs Flexibility and responsibility 29/06/2016CoreGrid Workshop - August 2010 20 In the Future…

21. www.egi.eu EGI-InSPIRE RI-261323 Grids have benefited from commoditisation –Hardware: HTC & HPC affordable to all –Networking: GBs can be moved over WAN –Software: Open source software comes of age Impacts of commodity virtualisation… –For transactional models  The ‘Cloud’: A model based on compute not data –For large distributed data-oriented models  The emergence of true ‘function shipping’? 29/06/2016CoreGrid Workshop - August 2010 21 Can we learn from others?

22. www.egi.eu EGI-InSPIRE RI-261323 EGEE: –Demonstrated a production e-infrastructure EGI: –Provide a sustainable production e-infrastructure EGI.eu established in Amsterdam –Supported transition through EGI-InSPIRE Contact: director@egi.eudirector@egi.eu 29/06/2016CoreGrid Workshop - August 2010 22 Summary