Apache Airavata GSOC 2013

Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced Science Tools Target Community: Science Gateways Enabling & Democratizing Scientific Research

What does Apache Airavata do? Compose, manage, execute, and monitor distributed, computational workflows. Wrap legacy command line scientific applications with Web services. Run jobs on computational resources ranging from local resources to computational grids and clouds. Manage provenance data.

Workflow Interpreter Application Factory Message Box Regist ry Apache Airavata API Lorem ipsumLorem ipsum insolensinsolens p1p1 m5m5 duo duo x End Users Gateway Developer Scientific Applicati on Core Developer Computational Resources Apache Airavata

Apache Airavata Components ComponentDescription XBayaWorkflow graphical composition tool. Registry ServiceInsert and access application, host machine, workflow, and provenance data. Workflow Interpreter Service Execute the workflow on one or more resources. Application Factory Service (GFAC) Manages the execution and management of an application in a workflow Messaging SystemWS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events Airavata APISingle wrapping client to provide higher level programming interfaces.

Hi, I’m Nolram. I’m a computational physicist. I run computational experiments everyday This is how typically I run my experiments

Scientific Application Another Scientific Application First I collect my observed data And then pass data to my applications & get the result This is starting to become a very tiring task

How can I make this much simpler…? Logically, this is how my life would be made easier… Is it possible to automate this flow sequence without my guidance?

Scientists from many different fields face this problem everyday. The solution is to use a workflow-powered science gateway to manage the experiment online. What is a workflow you ask? Well, you just saw one in our previous animation…

We introduce Apache Airavata, a system capable of composing, managing, executing, and monitoring small to large scale applications and workflows Want to see how it works? A Typical Workflow

Apache Airavata I will handover my data & my experiment details (the workflow) to the Airavata server The Gateway Airavata will complete the experiment & return me the results Results Progress of the experiment … and while I wait for results, Airavata will notify me with progress updates of my experiment

Let’s look closely how Airavata manages workflows. The Gateway Results Experiment progress Apache Airavata

Let’s look closely how Airavata manages workflows. The Gateway Results Experiment progress

Airavata main has 4 components… The Gateway 1. Workflow Interpreter Steer the workflow execution 2. The GFac Steer science app executions & data transfers Workflow Interpreter GFac Message Box Registry 3. The Registry Defines the available applications & records all results of experiments 4. The Message Box Records the progress of the workflow execution

A Stable API for Airavata Apache Airavata End Users Gateway Developer Scientific Application Computational Resources

Application Registration UI Application Registration UI Application Developer A1 Airavata Service Interface (wraps client API) Airavata Service Interface (wraps client API) Service Map XML Airavata Server Service Map to AWSDL Web Based workflow composer Workflow Developer Get AWSDL Put XWF A2 A3 W1 W2 W3 Web Based Experiment Builder Experiment Builder Get WI’s E1 E2 E3 W4 Shred Workflow Inputs Launch Workflow Web Based Workflow Monitor Watch Progress M1 M2 Get Workflow Graph M3 Monitor Workflow

Goal of the project Design Web-Based interfaces for Airavata: – Application Registration – Workflow Construction – Workflow Execution – Workflow Monitoring Provide an opportunity for GSoC to understand Distributed System in action Scope for Research and Software Engineering papers

Data Model Application Description – User describes inputs and outputs of the application. – Currently this information is captured in Service Map Schema. – This schema is stored in Airavata Registry as XML. Also the schema utility generates a application service WSDL from this schema using the Airavata WSDL Generator.

Application Registration UI Application Registration UI Application Developer A1 Service Map XML Service Map to AWSDL Web Based workflow composer Workflow Developer Get AWSDL A2 W1 W2 XML Airavata Server API Launch & Manage Jobs Notify progress of job or workflow execution Real-Time Monitoring Messaging Subsystem Registry Execute & Manage Computations Workflow Interpreter Application Factory (Gfac) Applicatio n Desc Workflow

A peek at one of the cluster Interconnect Nodes

Scheduling ‘qsub’ batch jobs on the cluster worker node SGE MASTER node Queue-AQueue-BQueue-C A Slot 1A Slot 2B Slot 1C Slot 1C Slot 2C Slot 3B Slot 1B Slot 2B Slot 3 B Slot 1C Slot 1 C Slot 2A Slot 1B Slot 1C Slot 1  Queues  Policies  Priorities  Share/Tickets  Resources  Users/Projects JOB YJOB ZJOB XJOB UJOB OJOB N

System characteristics System status Resources Job policies Resources Resource Matching JOB User User policies Groups Roles Departments Projects SelectionScheduling

Simplified Gateway Architecture One time Gateway Community Setup Community Account Grid Certificate username, password Gateway Interface Gateway Server Compute Servers Gateway Authentication Fetch Community Credential Grid Proxy Job Submit or File Transfer request Output Proxy, Job Request Job Status, Output Step 0 Step 1 Step 2,3,,

ParamChemBioVLabGridChemDES VLAB UltraScan Apache Airavata 1.0 NSG ParamChemBioVLab CIPRESPOPLAR GridChemDES UltraScan VLAB… …… Apache Airavata 2.0 Apache Airavata 1.0 Apache Airavata 1.0 Apache Airavata 1.0 Apache Airavata 1.0 Apache Airavata 1.0