1. One Click Cloud Orchestrator (OCCO): bringing Complex Complete Infrastructures Effortlessly to IaaS Clouds Peter KACSUK, Adam VISEGRADI, Jozsef KOVACS and Márk GERGELY MTA SZTAKI

2. Common national pools of resources Developing an Open Science Backbone Developing an Open Science Commons GovernanceOperations Security Shared capabilities based on open standards Research Infrastructures and long tail of science Core capabilities Open Science Cloud (e.g., VM management, Data storage/access/disc overy) PID Service registry and marketplace Research platform built on top of shared capabilities plus community owned resources Data products, tools, scientific gateways, virtual labs Multi-level governance with community participation Local National European Federated operations and support Service desk Monitoring and accounting Capacity management Service level management Network of CSIRT Federated IdPs, Auth and Authz Management of different levels of assurance From Member States Capacity dedicated to large RIs Free pools for long tail researchers Both pubicly funded and commercial providers (all supporting open standards and no lock-in) Credit to Sergio Andreozzi

3. BOINC project server BOINC client The whole infrastructure should be created by 1 click (Number of BOINC clients is a parameter) BOINC client Build an institutional desktop grid for universities Required BOINC expertise Took several weeks to establish Goal: setup the desktop grid in ~10 minutes in the cloud Easy creation of institutional desktop grids Easy creation of institutional desktop grids Used in IDGF-SP project Used in IDGF-SP project Motivations for OCCO (Example 1) Univ. cloud (or EGI cloud)

4. SZTAKI Desktop Grid server Gateway (WS-PGRADE) To run PS applications on BOINC 3G BridgeBOINC project DC-API plugin Metajob plugin BOINC client Virtualisation by GBAC BOINC client Virtualisation by GBAC Univ. cloud or EGI cloud Build the BOINC infrastructure with a gateway to submit parameter sweep applications into BOINC Motivations for OCCO (Example 2)

5. SZTAKI Desktop Grid server Autodock Gateway 3G BridgeBOINC project DC-API plugin Metajob plugin BOINC client Virtualisation by GBAC BOINC client Virtualisation by GBAC Univ. cloud or EGI cloud Setting up an Autodock SaaS (gateway with BOINC infrastructure): Motivations for OCCO (Example 3) Biologist, chemist can use it without any learning curve

6. Using Autodock SaaS service

7. E-mail notification of the user

8. Admin page showing the setup of the service

9. E-mail notification of the user

10. The gUSE portal can be reached at http://dgdemo5.lpds.sztaki.hu/ http://dgdemo5.lpds.sztaki.hu/

11. Motivations for OCCO The creation of this 1-click infrastructure took several weeks -> we want a solution that enables the construction in several days There are many other use cases where the goal is to set up a complete infrastructure on demand in the cloud Such an infrastructure typically consists of multiple virtual machine applications These should be instantiated and managed in an automated way We need an offline description how such an infrastructure should look like Then the infrastructure can be instantiated automatically, or with a single click of a user. Hence the name: One-Click Cloud Orchestrator The infrastructure typically defined as a graph where nodes are infrastructure services. Automation possibilities in a cloud o Node instantiation Through an API or a UI o Node configuration management Via Chef, Puppet, etc. o Our goal is to extend these to infrastructure

12. Problems to be solved Complex, multi virtual machine applications need special care that are not offered by current infrastructure clouds o Users frequently must fine tune their virtual infrastructures manually to meet their applications needs (scaling, inter-vm dependencies, error resilience) o Users of multiple clouds are usually restricted to the use of a particular provider Users of complex, multi virtual machine applications currently need low level understanding of application subcomponents and clouds for: o Configuration and deployment of subcomponents in a cloud context (this is partially offered by configuration management tools like Chef or Puppet) o Such VMs must be crafted for the subcomponents that are capable to interface with other dynamically created VMs We are developing a tool for the EGI communities to provide easy-to-use management for multi VM applications

13. State of the Art feature\SystemJujuOneFlowCloud- FormationHeatSlipStreamOpsWorks "OneClick" plans "OneClick" “vision” 31-Jul-1431-Jul-15 OS SupportUbuntuHypervisor dependant Fixed list of Linux distributions Windows Server FreeBSD Hypervisor dependant Supports wide variety of OS Amazon Linux, Supports wide variety of OS Ubuntu 12.04 LTS Supported cloud backends AWS EC2, HP Cloud Services, Windows Azure, Openstack OpenNebulaAWS EC2OpenStack OpenStack, AWS EC2, OCCI, Microsoft Azure, etc. AWS EC2 General EC2, All widely accepted Cloud Interfaces could be easily extended later OCCI, could be easily extended later Node management method Service ComposerImage BasedService Composer Image BasedService Composer Service Composer, Service Composer, Chef, other SotA composer(s) Generic Service Composer, Integrated Abstract image management Chef Open sourceYes NoYesPartlyNoYes (?) Hosted serviceHostedNoHostedNoHosted No No + SaaS Infrastructure Auto healing NoYes NoYesHopefullyYes Manual Infrastructure Scalability Yes NoYes Automatic Infrastructure Scaling NoYes NoYesNoYes UICLI, GUIWeb frontend, CLI management console, CLI, API CLI, API, Horizon dashboard Web UI management console, CLI, SDK API, CLI API, CLI, HTML5 Web frontend API, CLI, HTML5 Web frontend, SaaS One Click UINo Yes Service composer support Juju—Chef, Puppet —Chef loose integration with Chef loose integration with Chef, other SotA composer(s) Generic Service Composer, Integrated Abstract image management Multi-cloud provisioning YesNo YesNoYes Maturity (subjectively between 1-5) 315221124

14. OCCO in general Instead of basic components like virtual machines, OCCO can provide complete infrastructures of services in an on-demand, self-service fashion Automatic scaling and error recovery will be included. OCCO aims both providing o a high-level service with UI o exposing the underlying architecture as a framework to be built upon OCCO orchestrates o resource provider backends (e.g. IaaS clouds) o and configuration manager backends (e.g. Chef) based on statically defined infrastructure descriptions (“infrastructure-as- code” paradigm). The OCCO framework is developed to be highly versatile and backend-agnostic.

15. One Click Cloud Orchestrator (OCCO) Analogous to IaaS clouds: users can simply request complex infrastructures via a simple UI/API. o Instead of choosing from virtual machine types, users choose from virtual infrastructure types o They still face the same pay as you go model, and they can still utilize the flexibility of the IaaS’s just on larger scale. OCCO Node resolution OCCO

16. View of the OCCO Administrator 16 OCCO Administrators are expected to create node definitions and infrastructure descriptions (templates) o Based on existing configuration management descriptions (e.g. Chef recipes) o The templates are stored inside the OCCO provided template store in order to allow reuse

17. View of the OCCO User 17 Can browse and customize offered infrastructure description templates Can receive notifications about the state of his/her infrastructure (to be implemented) o Through email or an automated service (allowing immediate use after creation or changes) o If a change is applied to the template then the new updated deployment descriptor can be pushed to OCCO internals

18. Internal Architecture 18 Automated Infrastructure maintenance o infrastructure descriptor processing and VM management initiator Infrastructure Processor o internal depiction of a virtual infrastructure Cloud Handler o abstracts IaaS functionality (e.g., VM creation) for federated and interoperable use of clouds Service Composer (vm reshaper) o ensures awaited functionalities for VMs Information Dispatcher o de-couples the information producer and consumer roles across the architecture

19. Definition of an infrastructure

21. VM1 Cloud-init Deployment example: Wordpress-mysql MySQL CHEF OCCO VM2 Cloud-init Word press EC2 Register MySql Register Wordpress Deploy Wordpress Deploy MySQL Instantiate new VM with MySQL ipaddress Instantiate new VM Get MySQL ipaddress Infrastructure description

22. Development environment of OCCO In alpha stage Python 2.7 Git [https://gitlab.lpds.sztaki.hu/groups/cloud-orchestrator]https://gitlab.lpds.sztaki.hu/groups/cloud-orchestrator Jira [https://jira.lpds.sztaki.hu/browse/OCD] – agile sprint dev.https://jira.lpds.sztaki.hu/browse/OCD Documentation [http://c153-33.localcloud/util-doc/util.html]http://c153-33.localcloud/util-doc/util.html o Sphinx, docstrings: automatic documentation generation Testing: Python nosetests: automatic testing Deployment: setuptools o Development package repo: [http://c153-86.localcloud:8080/packages]http://c153-86.localcloud:8080/packages o Works best with virtualenv Package dependencies o PyYAML, argparse, python-dateutil, pika (for RabbitMQ), Boto, etc. Will be open-source after first release in 2015 Q3

23. Documentation of OCCO

24. Conclusions Prototype version of OCCO is working First version of OCCO is close to be released Short term goal: to create OCCI plugin for EGI FedCloud New use cases are continuously developed We are looking for partners to realize their use cases