1. INDIGO – DataCloud PaaS activities @INFN
RIA
INFN-Bari

2. Paradigm shifts
From static partitioning to dynamic resource allocation
Improve resource utilization
Improve scalability and fault tolerance
From Monolithic App to Microservices
Easier to develop
Easier to scale
Fault isolation
From VMs to Containers
Run your application (even legacy) everywhere: on public and private clouds
INDIGO-DataCloud RIA

3. Docker ecosystem in Openstack
Heat
Solum
Kubernetes
Docker
Magnum
OpenShift
CloudFoundry
INDIGO-DataCloud RIA

4. Heroku-inspired frameworks
Deis builds upon Docker and CoreOS to provide a lightweight PaaS
Build/Release/Run
Similar solutions:
flynn, tsuru
INDIGO-DataCloud RIA

5. Mesosphere
Mesosphere offers a layer of software that organizes your machines, VMs, and cloud instances and lets applications draw from a single pool of intelligently- and dynamically-allocated resources, increasing efficiency and reducing operational complexity
Key Concept: Data Center Operating System
INDIGO-DataCloud RIA

6. Mesos Two-Level Scheduling
INDIGO-DataCloud RIA

7. Mesos Frameworks Long Running Services Big Data Processing
Aurora, Marathon, Singularity, SSSP
Big Data Processing
Cray Chapel, Dpark, Exelixi, Hadoop, Hama, MPI, Spark, Storm
Batch Scheduling
Chronos, Jenkins, JobServer, Torque
Data Storage
Cassandra, ElasticSearch, Hypertable
INDIGO-DataCloud RIA

8. Cluster configuration - example
INDIGO-DataCloud RIA

9. Conclusions Preliminary tests Deis: Mesos:
Installation is simple, but takes too long (~1h for 5 nodes cluster)
Poor stability
Debugging not so easy
Supported apps: twelve-factors, only ONE HTTP port
Mesos:
Easy to deploy
Packages available for Debian/Ubuntu and Redhat/Centos
Automatic provisioning available
Easy to scale
INDIGO-DataCloud RIA

10. Backup Slides
INDIGO-DataCloud RIA

11. Service Discovery
Mesos-DNS (alpha) allows to connect to an application in the Mesos datacenter knowing only its name
can be made fault-tolerant by launching with a framework like Marathon, that can monitor application health and re-launch it on failures.
Marathon SD allows easy network communication between services
Based on HAProxy TCP/HTTP load balancer plus an assistant script that uses Marathon’s REST API to periodically re-generate a HAProxy configuration file
INDIGO-DataCloud RIA

12. Applications in Marathon
JSON app definition
curl -X POST -H "Content-Type: application/json"
Marathon hands over execution to Mesos.
Mesos executes each task in its own sandbox environment
Marathon knows how to handle application resources
A typical pattern in the development and deployment cycle is to have your automated build system place the app binary in a location that's downloadable via an URI.
Marathon can download resources from a number of sources, supporting the following URI schemes:
file:
http:
https:
ftp:
ftps:
hdfs:
s3:
INDIGO-DataCloud RIA