SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI MTA SZTAKI background for the DARIAH CC Zoltan Farkas MTA SZTAKI LPDS, Hungary

Outline The Laboratory of Parallel and Distributed Systems WS-PGRADE/gUSE Projects/solutions building on WS- PGRADE/gUSE Future plans Collaboration with HUBzero

Some LPDS-related projects Portal: – SCI-BUS – CloudSME – VIALACTEA – agINFRA Workflow: – SHIWA – ER-Flow Desktop Grid: – EDGeS – EDGI – IDGF(-SP) Cloud: – CLAKK – CloudSME

WS-PGRADE/gUSE overview Based on Liferay General purpose Workflow-oriented gateway framework Supports the development and execution of workflow-based applications Supports the fast development of domain- specific gateways by customization methodologies Most important design aspect is flexibility

Who is gUSE for? Power users (Workflow Developers) Develop WF applications Publish the completed WF applications for end-users End-users (e-scientists) Execute the published workflow (WF) applications with custom input parameters by creating application instances using the published WF applications as templates. SG Instance Developers Develop application domain specific SG instance SG Framework Developers Develop generic SG framework Increasing number of members

Workflows are DAGs Graph: – Big boxes: jobs – Small boxes: input or output data – Arcs: data dependency Concrete workflow: all properties necessary for execution are set Template: some properties of the concrete workflow are fixed

Flexibility in exploiting parallelism Each job can be a parallel program – Parallel execution inside a workflow node – Parallel execution among workflow nodes Multiple jobs run parallel – Parameter study execution of the workflow (or some nodes of the workflow) Multiple instances of the same workflow with different data files

WS-PGRADE/gUSE customization possibilities End User View Application Specific Module: ASM API Remote API

WS-PGRADE/gUSE for the DARIAH CC WS-PGRADE Application-specific interfaces App Catalog App Catalog Storage Bridge (Data Avenue) Storage Bridge (Data Avenue) Compute Bridge (DCI Bridge) Compute Bridge (DCI Bridge) Clouds Amazon AWS EGI FedCloud OpenStack OpenNebula Clouds Amazon AWS EGI FedCloud OpenStack OpenNebula Clusters PBS LSF Clusters PBS LSF Grids gLite ARC UNICORE XSEDE Grids gLite ARC UNICORE XSEDE Federated AuthN Federated AuthZ (HEXAA) Federated AuthZ (HEXAA) Seismology Proteomics Astrophysics agINFRA...

Swiss systems biology community science gateway (ETH Zurich) Gateway for one of the largest Proteomics research groups in Europe (Ruedi Aebersold, ETH) – ~100 users, ~100 TB of data – 6 Workflows, 4 portlets, thousands of datasets have been processed

Shahand, Benabdelkader, Jaghouri, Huguet, Al Mourabit, Caan, van Kampen.Olabarriaga. A Data-Centric Science Gateway for Computational Neuroscience. Journal of Concurrency and Computation Practice and Experience, accepted. Academic Medical Center gateways Large-scale Analysis of Biomedical Data NeuroscienceVirtual screening Common architecture Since ,000 comp hours (~3 years) 800+ workflow runs

Statistical Seismology Science Gateway Provides seven statistical seismology functions to international seismology community with three service levels: – Simple: Simplified GUIs – Advanced: Powerful programming interface – Expert: Elastic with embedded workflows Based on gUSE with ASM and Remote API Various methods/models from “integration of multi- source data” to “generation of hazard maps” seismo.ceng.metu.edu.tr

agINFRA Workflow Gateway Drupal module for harvesting Harvest metadata from Open Access databases (OAI2.0) Converts them with a WS-PGRADE workflow submitted using the Remote API

EU FP7 agINFRA project

Results  agINFRA workflow gateway

agINFRA Science Gateway & aggregator for CIARD RING

Main objective: Establish an agricultural knowledge centre and decision support system based on data gathered by an innovative, complex sensor system and from international open repositories relying on big data, cloud, and HPC technologies to support precision agriculture. Duration: Budget: 2.4 Mrd HUF (appr. 8 MEUR) URL: Consortium: Agrodat.hu: Big Data solutions to support precision agriculture LocalSensorGlobal

gUSE based gateways 18 Close to 130 deployments world- wide More than downloads from SourceForge

Summary WS-PGRADE/gUSE offers a complete science gateway technology based on Liferay for running feature-rich workflows on diverse set of compute resources, processing data from different types of data sources The customization methodologies offered enable easy development of application-specific interfaces, but also enable workflow execution from existing interfaces WS-PGRADE/gUSE is used by many different communities around the world, used not only for scientific data processing but teaching as well