Daniele Lezzi Execution of scientific workflows on federated multi-cloud infrastructures IBERGrid Madrid, 20 September 2013

2 Outline Introduction –Objectives –The COMP Superscalar framework COMPSs Interoperability in the EGI Federated Cloud –The EGI Cloud federation model –COMPSs integration with EGI FedCloud Evaluation –The Modeller service –Testbed Results –Single request scenario –Multiple requests scenario Conclusions and Future Work

Introduction

4 Objectives 1.Enhance the COMP Superscalar programming framework to interoperate with the EGI FedCloud. 2.Optimize the Modeller biodiversity service in EUBrazilOpenBio. 3.Evaluate the service performance on a federated cloud environment. 4

5 StarSs CellSs SMPSs GPUSs GridSs ClearSpeedSs ClusterSs OmpSs ClusterSs Cluster Programmability/Portability – Incremental parallelization/restructure. – Focus in the problem, not in the hardware. – Top/down programming. – Supported languages: Java, C and Python. – “Same” source code runs on “any” machine (on Java) Optimized task implementations Performance (Intelligent Runtime) Asynchronous (data-flow) execution and locality awareness. Automatically extracts and exploits parallelism. Malleable, matches computations to specific resources on each type of target platform. The COMPSs Framework

6 The COMPSs Framework (Programming Model) Resource 2 1. Identify tasks main program { } 2. Select tasks taskA(...); taskB(...); task selection interface { } taskA taskB Task Unit of parallelism...‏ taskA Asynchrony taskB Resource 1 Resource N 6

77 The COMPSs Framework (Interoperability)

COMPSs Interoperability in the EGI FedCloud

9 Standards and validation: emerging standards for the interfaces and images (OCCI, CDMI, OVF). Resource integration: Cloud Computing to be integrated into the existing production infrastructure. Heterogeneous implementation: no mandate on the cloud technology. Provider agnosticism: the only condition to federate resources is to expose the chosen interfaces and services. The EGI Federation Model

10 COMPSs integration with EGI FedCloud COMPSs Application: implementation of the application logic, where some tasks will be instrumented by the COMPSs runtime and executed remotely on EGI FedCloud resources. Cloud Connectors: implements a common interface allowing the resource management on an specific provider. OCCI Connector: translates COMPSs resource management calls to OCCI operations. Configuration: each provider’s available templates and images are set up on COMPSs configuration.

11 Retrieve AC With VOMS Proxy Cert. OCCI Account Synchronization COMPSs integration with EGI FedCloud Configure of COMPSs runtime (available cloud providers, endpoints, etc.) Configure of the OCCI connector (VM templates, VOMS credentials). Generate of the VOMS proxy certificate. Execute the COMPSs application. During the execution, some tasks could have hardware constraints (CPU, Mem, Disk, …) OCCI Connector: maps each task requirements to a suitable template of the available cloud providers, starting new VMs if needed.

12 COMPSs integration with EGI FedCloud A COMPSs VM is available with the required software in the VM repository. The EGI Marketplace contains the list of providers offering this VM.

Evaluation

14 Ecological niche: “Set of ecological requirements for a species to survive and maintain viable populations over the time.” (Grinnel, 1917) Species occurrence points Environmental variables Modelling algorithm Projected niche model The Modeller Service (Ecological Niche Modelling)

15 The Modeller Service (operations) –Convert: Converts multi-job request in a single requests. –Model: models a specie distribution for a given request. –Test: Checks the accuracy of the distribution. –Project: Project each distribution over geographical layers in raster format. –Translate: Translates the raster projection into an image.

16 The Modeller Service (evaluated scenarios) Single multi-job request (workflow) Multiple requests Single request scenario: Issues a multi-job request (6 species and 2 modelling algorithms) producing 12 distribution models. It exploits different FedCloud templates. Multiple requests scenario: tests the global service performance for a given workload pattern (Gaussian random) by issuing many requests with low complexity (3 tasks). ENM Service (OMWS2)

17 The Testbed –Client: Issues requests to the Modeller Service. –WS Server: Hosts the service logic and the COMPSs runtime. –EGI FedCloud: Composed of two providers, both providing rOCCI server and OpenNebula cloud middleware. –CESNET (Czech Rep.): Provide the service with XLarge VMs (4 cores and 15 GB of mem). –CESGA (Spain): Provide the service with Large VMs (2 cores and 8 GB of mem).

Results

19 Results: Single request scenario System load vs. available virtual resources

20 Results: Multiple request scenario 1.Workload Distribution: JMeter generates a random Gaussian workload. Generate 30 requests randomly distributed on a 0 – 2 minute frames. Each request issues low computation time requests. 2.Test phase: Test duration = 5 hours. Test starts with pre-created VMs :  1 Large (CESGA)  2 XLarge (CESNET) 3.Analysis of service key performance indicators (KPI): Response Time Dynamic Resource Consumption Throughput Evaluation of global service performance: …

21 Results: Multiple request scenario Workload Distribution:

22 Results: Multiple request scenario Response time vs. Resource Consumption:

23 Results: Multiple request scenario System throughput:

Conclusions and Future Work

25 COMPSs, programming framework for optimization of complex workflows on different infrastructures: Grid Clusters Clouds Extensions of COMPSs for the interoperability with the EGI FedCloud: Dynamic VM multi-provider management through the rOCCI connector. Support to x509 security. The results demonstrate that the runtime is able to: Serve multiple requests properly managing the pool of resources. Keep the performance independently on the workload. Optimize the resources consumption. Conclusions

26 Improve the connector to support contextualization features. Enhance resource management to support multi-core tasks. Future Work

Thank you!

28 Results: Multiple request scenario