1. Scientific Applications with HyperFlow and Scalarm on the PaaSage Platform
Marian Bubak
Department of Computer Science
AGH University of Science and Technology
Krakow, Poland
ESOCC 2016 Conference, Vienna, September 5-7, 2016

2. Coauthors Maciej Malawski Bartosz Balis Kamil Figiela Maciej Pawlik
Dariusz Krol
Renata Slota
Michal Orzechowski
Jacek Kitowski
Dennis Hoppe
dice.cyfronet.pl

3. Outline Motivation: scientific applications vs cloud resources
Cloud platform – PaaSage
Application deployment and execution modeling with CAMEL
Workflows on clouds with HyperFlow
Scalability and scheduling of workflows
Parameter study on clouds with Scalarm
Sample results
Conclusions

4. Scientific applications vs cloud resources
Cloud = a complex ecosystem
Users, advanced middleware services, security requirements, resource usage quotas, etc.
Challenge: a solution for deployment and execution of scientific applications on clouds in an automated cost- and performance-effective way:
adaptable to diverse cloud infrastructures
transparent to the user
lightweight: minimizes user’s effort (setup, configuration)
maintainable: easy to integrate and fast to update
leverage cloud elasticity for autoscaling of scientific workflows
loosely coupled integration with cloud management platforms
deployment description in a cloud independent way
automated deployment in cross-cloud environments
cost- and performance-based deployment optimization
scaling out/in based on application-specific metrics
Maciej Malawski, Bartosz Balis, Kamil Figiela, Maciej Pawlik, Marian Bubak: Support for Scientific Workflows in a Model-Based Cloud Platform. UCC 2015:

5. HLRS Molecular dynamics
Docking task is a parallel MPI job
either a 32+ core VM
Or multiple VMs (virtual cluster)
Compute intensive workflow
e.g. on 16 cores, 8M molecules, simulation time 0.05: Run time 50 minutes
Initialization
Preprocessing
MD simuation
Postprocessing
End

6. Montage (astronomy) workflow
Characteristics:
all tasks are "dataflow"
all consume and produce files
each task is fired only once
tasks are I/O intensive
Execution model:
HyperFlow passes the command to be invoked on a VM for each task
Executor fetches tasks ready for execution from a queue, executes the command, and puts a message back on the queue
Very large (100K+) pipelines (DAGs) of resource-intensive tasks

7. eScience workflows on the PaaSage platform
PaaSage community e.g. MD from HLRS
PL-Grid community: bioinformatics (genomics, proteomics)
metals engineering (complex metallurgical processes)
Virtual Physiological Human (Taverna and DataFluo workflows)
multiscale applications: fusion (Kepler workflows)
military mission planning support (EDA)
astronomy (Pegasus workflows)
Results
HyperFlow: workflow execution engine based on REST paradigm
Scalarm: massively self-scalable platform for data farming

8. PaaSage Platform http://www.paasage.eu
An open and integrated platform with an accompanying methodology that allows model-based development, configuration, optimisation, and deployment of cloud applications independently of underlying cloud infrastructures.

9. Model-based development and deployment of cloud applications
PaaSage cloud platform:
CAMEL: Cloud Application Modeling and Execution Language
Deployment model: components, connections
Requirements model
Scalability model
Multi-cloud application deployment
Autoscaling, adaptation
Integration with workflow systems:
CAMEL app model is generated from a workflow description
Workflow monitoring information triggers workflow autoscaling

10. Scientific applications on multi-clouds
Extensions to PaaSage platform
a new Domain-Specific Language for describing workflows
workflow planning based on user-provided objectives and constraints
a new workflow execution engine
Take benefit of PaaSage services
automatic and adaptive deployment of workflow components
autoscaling driven by rules
Workflow description (DAG)
Reasoner
Prepares a deployment plan
Workflow Engine
Executes workflow tasks
Deployer
Launches VMs
Constraint, objectives
Executionware
Deployment plan, autoscaling rules
Enforcement engine
Applies rules
Adapter
Concrete deployment commands and rules
Monitoring events
Upperware
Workflow Planner
Plans workflow execution
CAMEL description of workflow application
Bartosz Balis, Kamil Figiela, Maciej Malawski, Maciej Pawlik, Marian Bubak: A Lightweight Approach for Deployment of Scientific Workflows in Cloud Infrastructures. PPAM (1) 2015:

11. Lightweight workflow programming and execution environment
Simple wf description (JSON)
Advanced programming of wf activities (JavaScript)
Running a workflow – simple command line client hflowc run <workflow_dir>
function getPathWayByGene(ins, outs, config, cb) {
var geneId = ins.geneId.data[0],
url = ...
http({"timeout": 10000, "url": url },
function(error, response, body) {
...
cb(null, outs);
}); }
<workflow_dir> contains:
File workflow.json (wf graph)
File workflow.cfg (wf config)
Optionally: file functions.js (advanced workflow activities)
Input files

12. HyperFlow on the PaaSage platform
PaaSage cloud platform:
Model-based development of cloud applications
Multi-cloud application deployment
Autoscaling according to application demands
Integration of HyperFlow with PaaSage:
CAMEL app model generated from HyperFlow Wf description
Task scheduler delivers initial deployment plan and scalability rules
Workflow monitoring information triggers workflow autoscaling
Bartosz Baliś, Marian Bubak, Kamil Figiela, Maciej Malawski, Maciej Pawlik, Towards Deployment and Autoscaling of Scientific Workflows with HyperFlow and PaaSage, CGW’14

13. Workflow generic scenario: HLRS MD with MPI
Worker
Storage
Workflow Engine (Master)
MPI Worker
MPI Master
MPI
App Bin
Job Exec
WWW
NFS
Rabbit
HyperFlow
Redis
Worker
MPI Worker
MPI Master
Flexiant [location: UK]
Each HyperFlow worker requires a set of machines - a virtual cluster consisting of MPI master and MPI worker
The VMs need to be on the same IP subdomain
Application Binaries
App Bin
RabbitMQ Server
Rabbit
Job Executor
Job Exec
HyperFlow Engine
HyperFlow
NFS Server
NFS
Redis DB
Redis
MPI Runtime
MPI
WWW
Web server 13

14. Example application goals and scalability rules
My workflow can run in parallel and can scale up to 8 VMs
I need to dynamically scale out my virtual cluster based on utilization:
if resource utilization of VMs > 90% for more than 3 minutes then add a new worker VM
if resource utilization of VMs < 10% for more than 3 minutes then terminate this worker VM
I prepared an execution plan with constraints:
My workflow consists of 2 stages:
For stage 1 I need 8 VMs; For stage 2 I need to add 8 more VMs
→ please monitor the WorkflowStage metric published by workflow engine and if WorkflowStage > 1 then add 8 worker VMs
All VMs need to be of the same type:
m3.xxxlarge on Amazon or 8-core VM on any provider (alternative)
I need 16 core-hours to run my workflow
Please find the cheapest deployment
Additional constraint: I have a quota on VM number:
Max 8 instances on OpenStack
Max 20 instances on Amazon
I need to terminate all workers when the workflow is complete
→ please monitor WorklfowExecutionState metric published by workflow engine and if WorkflowExecutionState == DONE then terminate all the workers

15. Scheduling and provisioning plan
Example workflow with 3 stages
Scaling rule: launch 7 VMs for stage 2
Scaling rule: terminate 2 VMs for stage 3
Tasks of stage 1 VM
Tasks of stage 2
Tasks of stage 3
Time
Maciej Malawski, Kamil Figiela, Marian Bubak, Ewa Deelman, Jarek Nabrzyski: Scheduling Multilevel Deadline-Constrained Scientific Workflows on Clouds Based on Cost Optimization. Scientific Programming 2015: : :13 (2015)

16. Parameter study with Scalarm on PaaSage platform
Scalarm orchestrates parameter studies and data farming experiments
PaaSage framework manages Scalarm deployment in cross-cloud environment
Scalarm is modeled (services, deployment and scaling) with the CAMEL language
Obtained solution is published in the PaaSage social network
D. Król, M. Orzechowski, J. Liput, R. Słota, and J. Kitowski. Model-based execution of scientific applications on cloud infrastructures: Scalarm case study, in: Proceedings of Cracow Grid Workshop, pp , 2014.
D. Król, R. Da Silva, E. Deelman and V. Lynch, Workflow Performance Profiles: Development and Analysis, accepted: The International Workshop on Algorithms, Models and Tools for Parallel Computing on Heterogeneous Platforms (HeteroPar'2016), Euro-Par 2016.

17. Main goal: execution of the same application with different input parameter values i.e. support different steps of a data farming/parameter studies process
input parameter space specification
application execution with different input parameter values
collecting results and analysis

18. Overview of results
Behavior analysis of security forces M. Kvassay, L. Hluchý, S. Dlugolinský, B. Schneider, H. Bracker, A. Tavčar, M. Gams, M. Contat, L. Dutka, D. Król, M. Wrzeszcz, J. Kitowski, A Novel Way of Using Simulations to Support Urban Security Operations. COMPUTING AND INFORMATICS, 34(6), 2015.
Molecular dynamics - nano droplet simulation D. Król, M. Orzechowski, J.Kitowski, Ch. Niethammer, A. Sulisto, A. Wafai, A Cloud-Based Data Farming Platform for Molecular Dynamics Simulations, in: proc. 7th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), 8-11, London UK, IEEE 2014, pp. 579 – 584.
Hot rolling mill design D. Król, R. Slota, J. Kitowski, L. Rauch, K. Bzowski, M. Pietrzyk, Model-based approach to study hot rolling mills with data farming, in: T. Claus, et al. (eds.), Proc. of 30th European Conf. on Modelling and Simulations, Regensburg, 2016, OTH Regensburg 2016, pp
Sensitivity analysis D. Bachniak, J. Liput, L. Rauch, R. Słota, and J. Kitowski. Massively Parallel Approach to Sensitivity Analysis on HPC Architectures by using Scalarm Platform. In Parallel Processing and Applied Mathematics : 11th international conference, PPAM 2015 : Kraków, Poland, September 6–9, 2015 : book of abstracts,, page 93, 2015.
Material science - molecular dynamics + neutron scattering intensity calculations D. Król, R. Silva, E. Deelman, V. E. Lynch, Workflow Performance Profiles: Development and Analysis, accepted at Euro-Par 2016 (HeteroPar’16).

19. Summary HyperFlow + PaaSage improve reproducibility:
CAMEL: complete description of infrastructure
HyperFlow JSON: complete description of application
On-demand deployment of the workflow runtime environment as part of the workflow application
Workflow engine as another app component driving the execution of other components
Avoidance of tight coupling to a particular cloud infrastructure and middleware

20. https://github.com/dice-cyfronet/hyperflow
More at

21. Acknowledgements
This research was supported by EU FP-7 ICT Project PaaSage – Polish Grant 3033/7PR/2014/2