Ocean Observatories Initiative OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Scientific Workflows for OOI Ilkay Altintas Charles Cowart Daniel Crawl Stephen Henrie Michael Meisinger SDSC UCSD

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Scientific Workflows Support the Scientist Conceptual SWF Executable SWF From “Napkin Drawings” to Executable Workflows Fasta File Circonspect Average Genome Size Combine ResultsPHACCS

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Why Scientific Workflows? Important Attributes to Facilitate and Accelerate Research Assemble complex processing easily Access transparently to diverse resources Incorporate multiple software tools Assure reproducibility Build around community development model

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Workflows are a Part of Cyberinfrastructure Workflow Design Reporting Workflow Monitoring Workflow Execution Workflow Scheduling and Execution Planning Run Review Provenance Analysis Deploy and Publish Accelerate Workflow Design and Reuse via a Drag-and-Drop Visual Interface Facilitate Sharing Schedule, Run and Monitor Workflow Execution Promote Learning Support for end-to-end computational scientific process

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Ptolemy II Full list of contributors, projects, individuals and funding info are at the Kepler website!! Cross-project collaboration Initiated August 2003 Kepler 2.4 release: Scheduled for 01/2013 Kepler is a Scientific Workflow System

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 A Scientific Workflow in Kepler A green box is called an ‘actor’, which performs a task. This special actor represents an annotation component, such as BLAST search. Data flow is divided. Workflow parameters, which can be specified by users in the portal, are passed to workflow components.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Workflows in ION Support OOI operations Interactive QC Real-time data product generation Event-detection Visualization and data assimilation Adaptive sampling and resource command & control Support OOI end users Reuse and share templates of workflows for easy customized workflows R3 Development and integration strategy The AS subsystem provides the workflow framework (based on R2 capabilities in COI, CEI, DM) Other subsystems develop specialized workflow templates and actors (e.g., for interactive QC) For the R3 product, we develop a set of data and instrument specialized workflow realizations as needed Users can add to the set of R3 workflows in operational system

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 OOI Workflow Landscape

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Integrating ION and Kepler

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Integration is accomplished through the ION Workflow Agent Provides a safe ‘sandbox’ for each workflow to run Manages streaming data into and out of Kepler Provides a standard interface for using workflows in ION Will provide extended status information in the future Demonstration: Integrating Kepler and ION

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 ION Capability Container ION Capability Container

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Instrument N ION Capability Container ION Capability Container

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Instrument N ION Capability Container ION Capability Container Instrument N publishes data into the system.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Instrument N ION Capability Container ION Capability Container Instrument N publishes data into the system.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container A Workflow Agent is started to process this data.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container The Workflow Agent subscribes to the published data.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container The Workflow Agent starts a Kepler instance and begins feeding the data to the workflow.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013

Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container The Workflow Agent starts a Kepler instance and begins feeding the data to the workflow.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container Kepler passes the filtered results back to the Workflow Agent.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container The Workflow Agent publishes the filtered data into system.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Client M Client M Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Client M Client M Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container Client M can subscribe to the newly published data stream to access the filtered data.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Client M Client M Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container Client M can subscribe to the newly published data stream to access the filtered data.

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Client M Client M Kepler Workflow Agent Workflow Agent Instrument N ION Capability Container ION Capability Container Kepler Multiple Workflow Agent / Kepler pairs can be started as needed, processing any given set of streams using any given workflow. Workflow Agent Workflow Agent Kepler Workflow Agent Workflow Agent

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 DEMO

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Thanks & Questions… Ilkay Altintas Charles Cowart Daniel Crawl Michael Meisinger Stephen Henrie