1. 1 Ilkay ALTINTAS - July 24th, 2007 Ilkay ALTINTAS Director, Scientific Workflow Automation Technologies Laboratory San Diego Supercomputer Center, UCSD Working with Kepler Workflows to Automate Steps in the Scientific Method

2. 2 Ilkay ALTINTAS - July 24th, 2007 The Big Picture: Supporting the Scientist Conceptual SWF Executable SWF From “Napkin Drawings” … … to Executable Workflows Source: GuideToENM.pdf at the workshop web link. http://reap.ecoinformatics.org/Wiki.jsp?page=RequirementsWorkshop2007 http://reap.ecoinformatics.org/Wiki.jsp?page=RequirementsWorkshop2007

3. 3 Ilkay ALTINTAS - July 24th, 2007 Anatomy of a Kepler Workflow Search Panel Atomic Actor Link Director Toolbar Drag and Drop Annotation

4. 4 Ilkay ALTINTAS - July 24th, 2007 Anatomy of a Kepler Workflow Composite Actor Atomic Actor

5. 5 Ilkay ALTINTAS - July 24th, 2007 Anatomy of a Kepler Workflow Parameter Run Controls

6. 6 Ilkay ALTINTAS - July 24th, 2007 Some False Myths about Scientific Workflows Scientific workflows are… –… scripts/visual programs Can be similar, but there’s value added –… just workflow engines There are other components making the design and execution of workflows more effective Search features, provenance tracking, semantic guidance, etc… –… the user interface User interface is to make workflow design and manipulation easier

7. 7 Ilkay ALTINTAS - July 24th, 2007 Kepler can be used as a batch execution engine.

8. 8 Ilkay ALTINTAS - July 24th, 2007 Efrat JEAGER-FRANK 1 Christopher J. CROSBY 2 Ashraf MEMON 1 Viswanath NANDIGAM 2 J. Ramon ARROWSMITH 2 Jeffrey CORNER 2 Ilkay ALTINTAS 1 * Chaitan BARU 1 1 San Diego Supercomputer Center, University of California, San Diego 2 Department of Geological Sciences, Arizona State University A Three Tier Architecture for LiDAR Interpolation and Analysis *Presenting author

9. 9 Ilkay ALTINTAS - July 24th, 2007 R. Haugerud, U.S.G.S D. Harding, NASA Point Cloud x, y, z n, … LiDAR Introduction Survey Process & Classify Analyze / “Do Science” Interpolate / Grid

10. 10 Ilkay ALTINTAS - July 24th, 2007 LiDAR Difficulties Massive volumes of data –1000s of ASCII files –Hard to subset –Hard to distribute and interpolate Analysis requires high performance computing Traditionally: Popularity > Resources

11. 11 Ilkay ALTINTAS - July 24th, 2007 A Three-Tier Architecture GOAL: Efficient LiDAR interpolation and analysis using GEON infrastructure and tools –GEON Portal –Kepler Scientific Workflow System –GEON Grid Use scientific workflows to glue/combine different tools and the infrastructure Portal Grid

12. 12 Ilkay ALTINTAS - July 24th, 2007 Lidar Workflow Process Configuration phase Subset: DB2 query on DataStar Portal Grid Subset Analyze moveprocess Visualize moverenderdisplay Interpolate: Grass RST, Grass IDW, GMT… Visualize: Global Mapper, FlederMaus, ArcIMS Scheduling/ Output Processing Monitoring/ Translation

13. 13 Ilkay ALTINTAS - July 24th, 2007 Lidar Processing Workflow (using Fledermaus) Subset Analyze moveprocess Visualize moverenderdisplay Arizona Cluster NFS Mounted Disk IBM DB2 Datastar NFS Mounted Disk d1 d2 (grid file) d2 d1 iView3D/ Browser Create Scene file Fledermaus sd

14. 14 Ilkay ALTINTAS - July 24th, 2007 Lidar Processing Workflow (using Global Mapper) Subset Analyze moveprocess Visualize moverenderdisplay Arizona Cluster NFS Mounted Disk IBM DB2 Datastar NFS Mounted Disk d1 d2 (grid file) d2 d1 Browser Get image for grid file Global Mapper

15. 15 Ilkay ALTINTAS - July 24th, 2007 Lidar Processing Workflow (using ArcIMS) Subset Analyze moveprocess Visualize moverenderdisplay Arizona Cluster NFS Mounted Disk IBM DB2 Datastar NFS Mounted Disk d1 d2 (grid file) d2 ArcSDEArcIMSArcInfo ArcIMS d1

16. 16 Ilkay ALTINTAS - July 24th, 2007 Lidar Workflow Portlet User selections from GUI –Translated into a query and a parameter file – Uploaded to remote machine Workflow description created on the fly Workflow response redirected back to portlet

17. 17 Ilkay ALTINTAS - July 24th, 2007 Render Map DB2 Spatial query Client/ GEON Portal NFS Mounted Disk ArcInfo Compute Cluster x,y,z and attribute raw data process output KEPLER WORKFLOW Map Parameters Grass Functions submit Parameter xml Create Workflow Description ArcSDE ArcIMS Map onto the grid (Pegasus) Grass surfacing algorithms: Spline IDW block mean … Download data Binary grid ASCII grid Text file Tiff/Jpeg/Gif ASCII grid LIDAR POST-PROCESSING WORKFLOW PORTLET

18. 18 Ilkay ALTINTAS - July 24th, 2007 Portlet User Interface - Main Page

19. 19 Ilkay ALTINTAS - July 24th, 2007 Portlet User Interface - Parameter Entry 1

20. 20 Ilkay ALTINTAS - July 24th, 2007 Portlet User Interface - Parameter Entry 2

21. 21 Ilkay ALTINTAS - July 24th, 2007 Portlet User Interface - Parameter Entry 3

22. 22 Ilkay ALTINTAS - July 24th, 2007 Behind the Scenes: Workflow Template

23. 23 Ilkay ALTINTAS - July 24th, 2007 Filled Template

24. 24 Ilkay ALTINTAS - July 24th, 2007 Example Outputs

25. 25 Ilkay ALTINTAS - July 24th, 2007 With Additional Algorithms

26. 26 Ilkay ALTINTAS - July 24th, 2007 ORB Sources: Real Time Sensor Data Source. ANTELOPE Orb Data Recorded. KEPLER CLIENT WEB SERVER Start Kepler Engine via a Browser Specify Contour Parameters Look at Contour Plots with Google Maps Applications

27. 27 Ilkay ALTINTAS - July 24th, 2007 To Sum Up The next three days: –Discussions on use cases with an eye on the workflow requirements –Discuss other possible workflows –Developers taking notes on Kepler implementation details of possible workflow steps Latest release: http://kepler-project.orghttp://kepler-project.org

28. 28 Ilkay ALTINTAS - July 24th, 2007 Ilkay Altintas altintas@sdsc.edu +1 (858) 822-5453 http://www.sdsc.edu Thanks! & Questions…