Trident Scientific Workflow Workbench eScience’08 Tutorial Nelson Araujo, Roger Barga, Dean Guo, Jared Jackson Yogesh Simmhan, Catharine van Ingen, Nitin Gautam Microsoft Research Joby Thomas and the development team Aditi Technologies

Wright State University MSR (Trident) Summer ‘09 Interns Eran Chinthaka Indiana University David Koop University of Utah Satya Sahoo Wright State University Matt Valerio Ohio State University

Overview of our presentation today Technical Content Introduction Feature Overview and Logical Architecture Deep(er) dive into select features with demos Roadmap to delivery Design Philosophy and Exit Strategy Leverage COTS WFMS, build only what is required Extensible and open, integrate with community tools Drive development from actual eScience requirements Deliver as open source accelerator to the community

Ocean Observing Initiative (OOI) Formerly the NEPTUNE project Workflow for Ocean Observatories, part of an “oceanographer’s workbench” Jim Gray Collaboration with Univ. of Wash & MBARI

PanSTARRs Workflow Requirements (Astronomy) One of the largest visible light telescopes Four unit telescopes acting as one One Gigapixel per telescope Survey entire visible universe in 1 week Catalog solar system, moving objects/asteroids ps1sc.org: Univ. Hawaii, Johns Hopkins, … Workflow Requirements Load/Merge Databases Execute on Clusters Monitor workflow execution Logging, Provenance, Faults

Pan-STARRS Load & Merge Workflows Sanity Check of Network Files, Manifest, Checksum Validate CSV File & Table Schema Create, Register empty LoadDB from template For Each CSV File in Batch BULK LOAD CSV File into Table Start Perform CSV File/Table Validation Perform LoadDB/Batch Validation End Detect Load Fault. Launch Recovery Operations. Notify Admin. Determine affine Slice Cold DB for CSV Batch Switch OUT Slice partition to temp For Each Partition in Slice Cold DB UNION ALL over Slice & Load DBs into temp. Filter on partition bound. Start Post Partition Load Validation Switch IN temp to Slice partition End Detect Merge Fault. Launch Recovery Operations. Notify Admin. Slice Column Recalculations & Updates Post Slice Load Validation Determine ‘Merge Worthy’ Load DBs & Slice Cold DBs

Trident Public Website Accessible today http://beta.research.microsoft.com/en-us/collaboration/tools/trident.aspx From January ‘09 http://research.microsoft.com/en-us/collaboration/tools/trident.aspx

Logical Architecture Features Building on Windows Workflow

Trident Logical Architecture Visualization Design Workflow Packages Management Studio Community Workbench Monitor Web Portal (myExperiment) Scientific Workflows Administration Archiving Desktop Windows Workflow Foundation Registry Management Browser Trident Runtime Services Publish-Subscribe Blackboard WF Execution Hosts Fault Tolerance Provenance HPC Scheduling Others Trident Registry Data Model (Data Agnostic Abstraction) Data Access SQL Server SSDS S3 Others

Trident Features Libraries of activities, services, and workflows Prepackaged activities and workflows out of the box and custom libraries Registry with rich sets of workflow meta data Versions Workflow packages Social annotations (myExperiment)

Trident Features Two programming interfaces to Trident Use Visual Studio to develop custom activities and workflows and import them to Trident Visually Compose Workflows No programming and scripting is required Drag and drop a workflow or an activity Subsections

Execution Service Local or distributed execution of workflows HPCS cluster Cloud services Interactive and non-interactive execution service Publishes events to subscriber services, such as tracking, provenance, and monitoring.

Workflow Monitoring Remote and local monitoring Workflow processing status Input and output parameters Data products Performance

Management Studio Administration of workflows and workflow scheduling Registry management Monitoring

What is Windows Workflow? Part of Microsoft’s .Net framework 3.0, 3.5, and upcoming 4.0 Activities Runtime Tooling Host Process (.exe, IIS, …) WF Runtime Extensions Tracking Persistence … Workflow Activity Library Tooling VS Designer VS Debugger Rehosted Designer

Windows Workflow Base Activity Library Composite Basic

Workflow Authoring

Trident Workflow Composer An End User Application for Editing, Executing, and Monitoring Scientific Workflows

What Differentiates Scientific Workflow? Composition goes through many iterations Data flow is a first class citizen Need an easy way to publish and share Provenance Runtime Evolutionary Adaptable to different computing environments

Trident Workflow Composer Data Options & Sharing Workflow Library Composition Space Activity Library

Composer Demo

Flexible Data Store And Some More Trident Registry Flexible Data Store And Some More

Trident Registry Motivation: Why a new registry system? Single “point of truth” of the system Facilitates state synchronization actions Catalog keeps track of computing resources and state Flexible Storage What is it? Flexible store mechanism Supports Microsoft and non-Microsoft store providers Supports local, client-server and cloud architectures Non goals Replacement for LINQ or ER Framework Reference Catalog Unified view of the resources Stores references to internal and external resources Flexible provider mechanism to abstract access to external resources

Trident Registry Registry Connections

Trident Registry Registry Management

Trident Registry Data Providers: Abstracting “What’s out there” Storage providers Provides abstraction to data structures stored in the backend No assumptions on how data was stored and related Implemented using “verbs” and “subjects” actions “Store object user with these properties” “Relate this user object with this service as its owner” “Delete namespace object” Data abstraction layer and code generation C# generated code provides shield and programming API C# code generator generates SQL catalog for perfect datacode match

Trident Registry Data Providers: Abstracting “What’s out there” Creating new providers Why would I create a new storage provider? Enable Trident to store / retrieve state from other platforms Enable Trident to store / retrieve state on other systems Enhance existing providers with new features and abstractions What it takes to create a new provider Create a new assembly (or add to an existing provider assembly) Create a new class derived from Microsoft.Research.eResearch.Connection Drop our new DLL into Trident folder

Creating a new Registry Provider DEMO

Trident Registry Storage vs References Use Cases Object Tracking Data and Process Discovery All workflow aspects are exposed in the storage schema Allows rich query of data, activities, parameters, etc Data Providers Abstraction layer to external references (similar to registry data storage) Enables user applications to benefit from unified model Simplifies development Enables fault tolerance for external resource sources Not every workflow need to worry about these details All data provider knowledge resides in the registry Pluggable and flexible

Trident Registry Provider API Managed (.NET) API Library of choice for interacting with Trident Registry Simplifies lots of data complexity Abstracts verbs and actions into an object model Access to all Trident Registry objects and relations No need for servers and services to operate (access the data backend directly) Faster, no extra hops. Direct data access. API Native Managed Web Services Managed API Native API Useful for non-managed applications and systems integration Similar to Managed (.NET) API in terms of performance and requirements But more limited (not a 100% feature match right now) Native Web Services API Recommended for non-Microsoft platform integration, e.g. Linux and Mac OS Requires a IIS web server and service configured Greater control over data and process, higher data security Only core objects and relationships are exposed right now Extra parsing and processing hop. Need to consider cluster and load and balancing solutions for high-performance scenarios Web Services

A Distributed Eventing Model For Workflow Trident Blackboard A Distributed Eventing Model For Workflow

The Workflow Runtime and Tracking Services WF workflows launch in a runtime context Runtime thread controls WF related threads Execution thread Built-in services Custom services Built-in services track workflow execution Workflow events Individual activity events Data updates

Trident Blackboard A distributed Pub/Sub model for workflow eventing Why? Tracking information needs to be shared across compute nodes Workflows are evolutionary and thus messengers require a pluggable interface Large message volume means that the message broker needs to be light-weight and fast

The Blackboard Message Titled name/value pair collection All values are strings Title and names can resolve against an ontology Structure Example ‘Collection Title’ ‘WF Runtime Event’ ‘name 1’ ‘name 2’ ‘name 3’ ‘value 1’ ‘value 2’ ‘value 3’ ‘Type’ ‘Job ID’ ‘Activity ID’ ‘Event Order’ ‘Activity Started’ ‘{ GUID }’ ‘NetCDF Reader’ ‘5’

The Blackboard Message Titled name/value pair collection All values are strings Title and names can resolve against an ontology Structure Example ‘Collection Title’ ‘WF Runtime Event’ ‘name 1’ ‘name 2’ ‘name 3’ ‘value 1’ ‘value 2’ ‘value 3’ ‘Type’ ‘Job ID’ ‘Activity ID’ ‘Event Order’ ‘Activity Started’ ‘{ GUID }’ ‘NetCDF Reader’ ‘5’ Publisher Workflow Tracker Subscriber Subscriber Database Logging Provenance Store

Blackboard Architecture Publisher Interface Subscriber Interface Trident Workflow Executor WF Runtime Services Publisher Blackboard Subscriber Publisher Subscriber Publisher Subscriber Message Subscription Information Lightweight Message Queue

Blackboard Architecture Message Routing Publisher Interface Message Rerouting Subscription Information Management Recovery Logic Subscriber Interface Trident Workflow Executor WF Runtime Services Publisher Blackboard Subscriber Messages Publisher Subscriber Publisher Subscriber Message Subscription Information Lightweight Message Queue

Blackboard Architecture Subscription Information Routing Publisher Interface Message Rerouting Subscription Information Management Recovery Logic Subscriber Interface Trident Workflow Executor WF Runtime Services Publisher Blackboard Subscriber Messages Publisher Subscriber Publisher Subscriber Subscription Information Message Subscription Information Lightweight Message Queue

Blackboard Architecture Internal Technologies Publisher Interface Message Rerouting Subscription Information Management Recovery Logic Subscriber Interface Trident Workflow Executor WF Runtime Services Publisher Blackboard Subscriber Messages Publisher Subscriber Publisher Subscriber Subscription Information Message Subscription Information Lightweight Message Queue Windows Workflow (WF) Windows Communication Foundation (WCF)

Blackboard Architecture Logging and Monitoring Example Publisher Interface Message Rerouting Subscription Information Management Recovery Logic Subscriber Interface Trident Workflow Executor WF Runtime Services Config File Tracking Blackboard File Writer Messages Composer Registry Resources ‘WF Runtime Event’ ‘Type’ ‘Job ID’ ‘Activity ID’ ‘Event Order’ ‘Activity Started’ ‘{ GUID }’ ‘NetCDF Reader’ ‘5’ Message Subscription Information Lightweight Message Queue

Blackboard Demo

Trident Tips and Tricks

Interoperability Story Silverlight execution environment Web frontend for management and execution Allows non-Microsoft operating system to use and admister Trident Interface with other systems Cove myExperiment

Interface Trident  Other Systems Integration with UW COVE system DEMO

Trident Tips and Tricks Productivity Tools Database ready activities Simplifies development of database aware workflows Code generator improves development productivity Data visualization and charting activities Web Service ready activities Simplifies development of web service aware workflows

Trident Roadmap to Release

Trident Road Map Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Composer framework Registry Distributed execution service Sprint 2 Service and Tray Icon (run workflows locally and remotely) Workflow model Open and Save workflows with Workflow Model Subsections Intermediate results IFELSE Workflow over workflow Sprint 3 FOR-LOOP and Replicator Property Sheets for workflows and activities Monitoring (WF events, input & output parameters, performance) Data products (input and output) Blackboard Logging PanStarrs workflow support Sprint 4 Invoke Web Service and DB stored procedures Workflow packages Provenance (PanStarrs) Registry Manager Administration Console and workflow scheduling Remote monitoring Sprint 5 Silverlight based Composer Trident Portal (myExperiment) Deployment topologies desktop and workgroup (same domain) Fault Tolerance