1. Grids, Clouds and the Community. Cloud Technology and the NGS Steve Thorn Edinburgh University Matteo Turilli, Oxford University Presented by David Fergusson

2. NeSC, University of Edinburgh National eScience Centre – Support and develop IT supported advanced research University of Edinburgh – College of Science and Engineering – Coordination and cooperation to make best use of leading research across research.

3. NGS JISC supported national service to: – Provide a framework for institutions to cooperate and share services. – Help promote access to JISC services and synthesis for researchers.

4. Views of the Clouds Infrastructure as a Service Platform as a Service Service as a Service Software as a Service Private Clouds Public Clouds Hybrid Clouds Cloud vs Web 2.0 (?)

5. Views of the Clouds Infrastructure as a Service Platform as a Service √ Service as a Service Software as a Service Private Clouds Public Clouds Hybrid Clouds Cloud vs Web 2.0 (?)

6. Cloud computing for NGS/NeSC (incomplete list) Dynamic provision of services – Individualisation – Peaky demand Extending virtualisation Packaging – Service configuration – Provenance for research – New publishing models

7. Head in the clouds? Dynamic (service) provisioning How is it applicable to the NGS? Training – Rapidly deploy NGS services for training – Isolate training from production Other – Specialised research environments – Rapid deployment Identify use cases and gather requirements

8. NGS 3 EWP2 “NGS Agile Deployment Environments” EPSRC funded, 2 years People – Matteo Turilli (OeRC, Oxford) [0.75 FTE] – Steve Thorn (NeSC, Edinburgh) [0.5 FTE] – David Fergussion (NeSC, Edinburgh) [WP Leader]

9. Overview Agile service deployment Virtualization vs. Cloud? Use cases and requirements gathering – Training – Identify other (scientific) communities Create images – NGS Services. Which ones?

10. Overview (cont.)‏ Realistic usage – Training event on virtualized infrastructure Hosting infrastructure? – Amazon EC2 compatible De facto standard currently, with open source implementation – Ease of deployment – Eucalyptus, Nimbus and others Hardware – Edinburgh: 8 cores ⇒ 16+ dual cores => 64 cores – Oxford: 64 cores (older)‏, new cluster => 64 cores

11. Eucalyptus “Elastic Utility Computing Architecture Linking Your Programs To Useful Systems” Open source and Commercial Amazon Web Services API compatible – EC2, storage - S3, Elastic Block Store (EBS)‏ Easy to install Xen and KVM hypervisors – Commercial version supports others (inc. VMWare)‏

12. In the past We have worked with Xen in the past to have Live CDs Virtualisation Works, but – Issues with security setups – networking

13. Eucalyptus networking challenges Eucalyptus 'Managed' networking – Different networking modes – 'Managed' is the most flexible and feature rich – more complex – Allows elastic IP pool and image isolation. VMs have private and public IPs – Introspection issues Elastic IP – User assignable (may be somewhat different from Amazon)‏ X509 Service certificates (NGS Host)‏ Switch configurations

14. cont..... Security Groups (EC2) – Implemented in Eucalyptus – isolate VMs VM public traffic routed through Cluster controller – Instance doesn't have knowledge of its public IP – Bit like a NAT Implications for GSI: $GLOBUS_HOSTNAME

15. Eucalyptus architecture Cloud controller – Entry point – Gathers information Cluster controller – Schedules VM execution – Manages virtual network Node controller – Controls VM execution (Xen running on node)‏  Storage controller (Walrus)‏ implements Amazon’s S3 interface

16. cont..... Security Groups (EC2) – Implemented in Eucalyptus – isolate VMs VM public traffic routed through Cluster controller – Instance doesn't have knowledge of its public IP – Bit like a NAT Implications for GSI: $GLOBUS_HOSTNAME

17. Clouds vs Virtualisation Similar security and networking issues in Clouds and Virtualised instances – Virtualisation – virtualise instance – Clouds – virtualise the network (and other things) too All arise from the requirements for rapid, automated, dynamic, reliable, reproducible, robust, provisioning

18. Progress Started with Eucalypus 1.4.2 Eucalyptus 1.6.2 deployed at Oxford and Edinburgh Existing Images: – GSI-OpenSSH server – 'Single node cluster': torque/maui + Globus GRAM & GridFTP Next step – some real world testing of phase1 images. Image snapshots – not straightforward

19. Further work Re-evaluate hosting infrastructure Develop more images – Distributed torque/maui cluster + GRAM & GridFTP – Condor & GRAM? – 'Core site' Training event in near future Identify pilot community & gather requirements Deploy fledgling cloud infrastructure

20. Further work 2 Can snap shotting in Clouds side step packaging issues (configuration) in middleware? – By automating the copying and re-deployment of successful server installs. – Not just having a machine but a set of services which can be copied and deployed directly

21. Statement 1 NRENs are still able to provide services that are generally better or more economic than those from commercial services providers. Commercial offerings are currently shaped to particular modes of usage (due to their “by-product” heritage. This may limit utility for some users. EBI have reported issues due to virtual network sharing & unpredictability. Data out costs are high Commercial providers can offer “infinite” resource rapidly – not easy for NRENs Should NRENs own large computing centres? Payment models in institutions & research – FEC in the UK

22. Statement 2 NRENs generally operate as a network for a closed group of users who have advanced requirements to support their research and education users. There are certainly major issues with current commercial provision in regards to data confidentiality, isolation of users, service/data guarantees, internal/external access. – Recent data losses from commercial services Commercial providers are almost certainly at an advantage in providing simple generic computing NGS/UoE sees need to provide more complex research services.

23. Statement 3 The NRENs do not compete with commercial ISPs, but offer a different level of service in parallel with them. This is necessarily so. NRENs and other academic providers should find “blue water” for services. – Trust networks and security for public clouds ? – Service discovery ? – Data services ? – Accounting, monitoring ? –............. > Commercial providers will have no interest in some research services, cf. current research computing software provision.