1. EGI Applications Database VM Operations Dashboard
Marios Chatziangelou, et al.
Institute of Accelerating Systems and Applications (IASA)

2. Capabilities
A community driven, central service that stores and provides:
software solutions (in the form of native software and/or virtual appliances), originated from almost every scientific area/discipline
reference of scientific datasets (pilot - under development)
the programmers and scientists responsible for them
the publications derived from the registered items (SW, VA & datasets)
enabling users to deploy and manage Virtual Machines to the EGI Cloud infrastructure through the VMOps dashboard

3. Software Marketplace Registry for Software items:
Applications, tools, Workflow frameworks and instances, Science Gateways, Middleware products, …)
Offers release management capabilities
- unlimited series of releases
- light-weight & collaborative, release
management process
Acts as a repository for binary artifacts
unlimited number of repositories per register software
generic tarballs, RPM & DEB (32bit/64bit) binaries
multiple flavor / operating system combinations
simplified, web-based, process for uploading the binary artifacts
YUM & APT repositories for automatic distribution
artifacts populated through the UMD Community Repository

4. Cloud Marketplace (1/2) Registry for virtual appliances (VA)
a logical container of versioned image file & metadata bundles
Registry for software appliances (SA)
a logical container of VA versions & contextualization scripts bundles
VA distribution medium
distributing endorsed VAs to the resource providers/sites
Resource providers catalogue
list of the VAs which are available by each site/resource provider
Virtual Organizations (VO) catalogue
list of the VAs which are available for each VO member

5. perform VM operations using
Cloud Marketplace (2/2)
Remote Sites
Search for vAppliances
Information System
(Top-BDII for
FedCloud sites)
cloud m/w
Get availability & usage details
Start/Stop/…. an image using rocci tool
local info system
push info
about the VA
availability
and usage
cloudkeeper
user::visitor
perform VM operations using
the AppDB VMOps
subscribe and/or get any update
AppDB
Cloud Marketplace
AppDB Image List Store
generate
image_list
AppDB portal
(Re)publish VO or Project’s accepted list of Virtual Appliances
Publish new vAppliance version
Insert/update/remove image(s)
Make notes for interaction
Submitter
- Through the AppDB portal (=graphical interface) is able to register a vApp
The vApp is versioned. Each vApp version contains VM images, The submitter can create a vApp version either using the AppDB portal (graphically) or even through a command line tool (mainly for automation i.e. upload updates of images without human intervention)
Can create vApp version and at the end publish it and therefore meke it publicly available
VO or Project Manager
Selects the vApps (or their updates) that considere as suitable to be offered by his/her sites of responsibility
And publishes the chosen ones in to the AppDB Image List Store
Sites / Resource prividers
Subscribe and download the vApps or their respective updates
And inform back the AppDB that indeed the sites are downloaded and are accepted by the sites (EGI is using the TopBDII for this but this part of the chain can be replaced by other information systems or technilogies)
The site does not only inform AppDB about the vApp availability but also gives back usage ditails (= all details the user needs in order to make use of the vApp in question. In EGI. usage details = occi_id, template_id site occi endpoint)
The end user
Through the AppDB portal can search for a vApp and perhaps download it for use it into his own PC (i.e. VirtualBox)
Gets the info about which vApp is available in wich site and how he//she can use it (again, In EGI that means => occi_id, template_id site occi endpoint).
Having the usage details he/she can perform brokering actions (start/stop/…) to the remote site by using the OCCI command line tool
Next in the raw: to give the ability to the end user to perform brokering action through the AppDB portal in a graphicall way. Hiding all the technicalities (occi_id, template_id site occi endpoint) .from his eyes
Select/update/exclude vAppliances
Create a new vAppliance version
Register a vAppliance
VO or Project Manager
AppDB portal
AppDB portal
user::submitter

6. VM operations dashboard
AppDB VM Operations dashboard
The objective (EGI-Engage DoW) : “The EGI Applications Database (AppDB) will evolve from its current role as catalogue of applications and virtual machines images (VMI) to include a graphical user interface allowing authorized users to perform basic VM management operations”
AppDB
Portal
(main service)
VM operations dashboard

7. The AppDB VMops dashboard (1/5)
Highlighted features for the end-user
User identification, including VO membership & role(s)
Provide user views of deployed topologies and VM instances
Personalized, wizard-like, topology builder and the necessary tools, for:
Create a new topology with one (or more) instances of a specific VM
Attach additional storage to the VM instances
Deploy/Un-deploy a topology
Start/Stop a topology (= all the VM instances of a topology)
Start/Stop a single VM instance
Configure VM (cloud-init & ansible)
Execute bash script on deployment time

8. The AppDB VMops dashboard (2/5)
Wizard-like topology builder
pick a Virtual appliance
select the VO for consuming the resources
choose a Site
specify the resources needed (ram, cpu, disk,….)
make the final adjustments (if any)
multiple instances
additional storage
additional users
on_deployment script
and…… launch

9. The AppDB VMops dashboard (3/5)
View & manage a topology
Switch between VMs within a topology
View details for each VM, like:
IPs
How to access (ssh command, generated/provided keys, users etc)
Attached Storage (incl. mount points)
Logs (general & ansible/contextualization)
OCCI ID for moving to CLI world
Manage topology (start/stop/undeploy) or individual VMs (start/stop)

10. The AppDB VMops dashboard (4/5)
List of topologies or VM instances
available filters:
active (not undeployed/failed)
undeployed by the user (you!)
undeployed by the infrastrucure (not by you!)
failed
all
All the details needed for each topology or MV instance

11. The AppDB VMops dashboard (5/5)
View of the infrastructure availability any point of time

12. The recipe ....
What it was actually needed for the AppDB VMOps to be realized?
Information &
Software (development)

13. Correlated data fetched from the external sources & AppDB DB
Information
Integrated with the main service [AppDB main service as a source of info]
Resource Provider/Site static data
GOCDB
AppDB Portal
(main service)
API
VMOps Dash
Downtimes
Which VM and where is available
Top-BDII
OCCI IDs, resource tpls & other
dynamic info
Argo
Sites monitoring values
Correlated data fetched from the external sources & AppDB DB
Ops Portal
List of VOs
VO membership & roles
Perun
Authentication & Authorization
EGI Checkin
VO membership & roles

14. Infrastructure Manager
Software blocks
High-level/Context diagram
Dashboard
(1 inst.)
User’s interface
User/accounts configuration
Information collector
Auth/AuthZ
VMOps
(clustered 4 inst.)
Action management
Monitoring & State management
Proxy certificate management
Infrastructure Manager
(clustered 10 inst.)
OCCI communication layer

15. Next steps
Implement workflows/functionality to support complex topologies
ability to use different types of VMs within a topology
example: 1 centos7 and 2 VMs of ubuntu in a single topology
interconnected VMs
example: the centos holds the database and the ubuntu VMs offers the content management system (CMS), i.e. WordPress, for load balancing
shared block storages among VMs
spread VMs in different sites (theoretically this could be also possible)
Advanced functions, like
Snapshotting
DNS on demand for the VM instances
Support of any VO
Moving from proxy certificates to OIDC
Support of native connectors (OpenStack & OpenNebula)

16. Technical features & details
Highlighted technical features
TOSCA compatible
Infrastructure Manager [1] as the OCCI communication layer
Rest API for integration with other services
Single-Sign-On with the portal and the rest of the AppDB sub-services
Scalable architecture (one front-end many back-ends for load balancing)
Strictly technical details
NodeJS/HTML5/CSS/Javascript
MongoDB/Mysql
RESTful API
[1].

17. VM Operations Dashboard
Summarizing ...
AppDB
Portal
(main service)
VM Operations Dashboard
vAppliance VM
...
Block storage(s)
topology
Software
EGI fedCloud
infrastructure
Datasets
Support of any Virtual Organization
Complex topologies
Distributed among sites
Interconnected
Shared block storages
Full monitoring of deployed topologies
Dynamic DNS support ……