Presentation is loading. Please wait.

Presentation is loading. Please wait.

Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal,

Similar presentations


Presentation on theme: "Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal,"— Presentation transcript:

1 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal, Ewa Deelman, Gaurang Mehta, Mei-Hui Su, Karan Vahi USC Information Sciences Institute Marina del Rey, CA pegasus.isi.edu

2 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Motivation Many workflow systems exists today The choice of particular system often dictated by who you know Various workflow system have different capabilities Application components versus services Visual vs. scripting workflow descriptions Performance optimization, etc. Can you combine two separate systems? What are the issues?

3 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Kepler (UCSD and UCDavis) Scientific workflow management system based on Ptolemy II Allows scientists to visually design and execute scientific workflows Actor-oriented model with directors acting as the main workflow engine Enables different models of computation.

4 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Pegasus (USC/ISI) Based on programming language principles Leverages abstraction for workflow description to obtain ease of use, scalability, and portability Provides a compiler to map from high-level descriptions to executable workflows Correct mapping Performance enhanced mapping Relies on a runtime engine to carry out the instructions Scalable manner Reliable manner

5 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Combing Kepler & Pegasus Integration of Kepler visual programming environment with the grid mapping abilities of Pegasus Giving Kepler users the ability to map their large workflows onto the grid Giving Pegasus users a visual workflow composition tool Differences in the level of abstraction of workflow description

6 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Kepler Provenance Challenge Workflow

7 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Concrete Workflow Generation and Mapping

8 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Implementation Strategy Develop Pegasus-specific entities Abstract jobs Directors and actors “Pegasus Director” and “Pegasus Jobs (Actor Entities)” act as the main grid components to execute a given grid computation Focus mainly on abstract jobs in the Kepler environment portable and resources-knowledge independent workflow descriptions

9 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Integration

10 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Pegasus Actor & Director Entities

11 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Visual Abstract Workflow Creation Users can create visual models of abstract workflows and specify logical transformations without specifying grid resources

12 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Job Abstract Configuration --Integration with the Transformation Catalog

13 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Resultant Abstract Job on Kepler Canvas: A Pegasus abstract job can take in multiple input files as can output multiple output file Grid resources information is not expected in such an actor.

14 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Support for Concrete Jobs --- useful for monitoring and debugging A concrete job requires specific grid resources information from the scientist. Allows the scientist to directly execute jobs on the grid

15 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Pegasus Director/ DAX Generator Controls the execution of all the job (actor) entities and creates a resulting directed acyclic graph in XML format Generates a DAX Gives it to DAGMan for execution

16 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Sample DAX Generated :

17 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu

18 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Provenance Challenge Workflow in Kepler/Pegasus In Kepler each node needs a unique name, so TC needs many duplicate entries

19 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Integration Benefit for Pegasus users Visualizing/ Debugging Existing Models: Support a scientist trying to redo/visualize or easily re-configure existing DAX Provide option to upload existing DAX files into the workspace Convert the specified DAX file into a MoML (Kepler’s) format by passing it through an XSLT processor and generating the required directors and actors on the canvas Issues of scalability (only small workflows can be visualized) Scoping may need to be applied

20 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Integration Issues Kepler acts a visual programming environment Actors represent single units of computation with data flow among each other Some configuration not intuitive (TC entries) There is no concept of representation of files separately in Kepler Have multiport I/O ports for each job The user is given the option to connect as many files going into and coming out of the port Potential use of integrated environment for debugging Not done Integration with Pegasus data registry No monitoring of execution in Kepler Use of Kepler’s workflow execution engine Support for Kepler actors in Pegasus

21 Ewa Deelman, deelman@isi.eduwww.isi.edu/~deelmanpegasus.isi.edu Relevant Links Kepler: http://kepler-project.orghttp://kepler-project.org Pegasus: http://pegasus.isi.eduhttp://pegasus.isi.edu DAGMan: www.cs.wisc.edu/condor/dagman/www.cs.wisc.edu/condor/dagman/ Provenance challenge: http://twiki.ipaw.info/bin/view/Challenge/ http://twiki.ipaw.info/bin/view/Challenge/ Workshop on Tuesday NSF workshop on Challenges of Scientific Workflows: www.isi.edu/nsf-workflows06www.isi.edu/nsf-workflows06


Download ppt "Ewa Deelman, Integrating Existing Scientific Workflow Systems: The Kepler/Pegasus Example Nandita Mangal,"

Similar presentations


Ads by Google