HUBzero® Platform for Scientific Collaboration Copyright © 2012 HUBzero Foundation, LLC International Workshop on Science Gateways, ETH Zürich, June 3-5, Bringing Scientific Workflow to the Masses via Pegasus and HUBzero Michael McLennan*, Steven Clark*, Ewa Deelman, Mats Rynge, Frank McKenna, Derrick Kearney* and Carol Song* *Purdue University, USC Information Sciences Institute, University of California, Berkeley

Copyright © 2012 HUBzero Foundation, LLC What is HUBzero? International Workshop on Science Gateways, ETH Zürich, June 3-5, HUBzero Open source software platform used to build… “hubs” “Science Gateways” or “Collaboratories” for Research, Education, and Collaboration License LGPL-3.0 Linux/Apache/MySQL/PHP

Copyright © 2012 HUBzero Foundation, LLC Hubs for many disciplines 540,063258,791nanoHUB.org 230,69859,175nees.org 59,65929,965GlobalHUB.org 32,70022,651pharmaHUB.org 32,31411,199vhub.org 37,4143,132HABRIcentral.org 15,8983,933cceHUB.org 14,9775,797ciHUB.org 12,3043,510StemEdHub.org 11,7541,907iemhub.org 10,9951,863C3Bio.org 7,3171,246cleerHUB.org visitorsusers ~1,000,000 visitors per year International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013

Copyright © 2012 HUBzero Foundation, LLC Where are the users? 1,000,000 visitors total for all hubs each year International Workshop on Science Gateways, ETH Zürich, June 3-5, = Browsing = Simulation

Copyright © 2012 HUBzero Foundation, LLC 5 Global community 27 Foundation, LLC  Non-profit organization  Independent owner of HUBzero code  Promotes dissemination and outreach  Sponsors HUBbub Conference  Coordinates software contributions International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013

Copyright © 2012 HUBzero Foundation, LLC What is Pegasus? International Workshop on Science Gateways, ETH Zürich, June 3-5, Ewa Deelman USC Information Sciences Institute Development Team: Gideon Juve, Rajiv Mayani, Mats Rynge, Karan Vahi Pegasus Workflow Management System (est. 2001)

Copyright © 2012 HUBzero Foundation, LLC Advantages of Pegasus International Workshop on Science Gateways, ETH Zürich, June 3-5,  A collaboration with U Wisconsin Madison / HTCondor  Used by a number of applications in a variety of domains  Provides reliability—can retry computations from the point of failure  Provides scalability—can handle large data and many computations (kbytes-TB of data, tasks)  Optimizes workflows for performance  Automatically captures provenance information  Runs workflows on distributed resources: laptop, campus cluster, Grids (DiaGrid, OSG, XSEDE), Clouds (FutureGrid, EC2, etc..)

Copyright © 2012 HUBzero Foundation, LLC Industrial Strength International Workshop on Science Gateways, ETH Zürich, June 3-5, Since Jan 2013

Copyright © 2012 HUBzero Foundation, LLC How does Pegasus work? International Workshop on Science Gateways, ETH Zürich, June 3-5, APIs for workflow specification Workflow Spec DAX = DAG in XML Executable Workflow (After Pegasus Planning)

Copyright © 2012 HUBzero Foundation, LLC Two Audiences Researchers Casual Users Use “workspace” (remote desktop) Launch simulation jobs on grid/cloud Share desktops Build and deploy tools Use “tools” (app store) Simple, intuitive interfaces Share tools International Workshop on Science Gateways, ETH Zürich, June 3-5,

Copyright © 2012 HUBzero Foundation, LLC Hub Web Server Execution Host 1 Execution Host 2... Submitting Your Pegasus Workflow International Workshop on Science Gateways, ETH Zürich, June 3-5, Pegasus “DAX” Workflow submit pegasus-plan --dax myworkflow.dax

Copyright © 2012 HUBzero Foundation, LLC Submit: Parameter Sweeps 12 If you normally run a command like this… spice3 –b circuit Submit to remote resources like this… submit –v DiaGrid spice3 –b circuit submit –p spice3 * common collector amplifier: mid- and low-frequency analysis.model m1 npn bf=350.width in=80 out=80 * * dc bias circuit r1 2 4 r k vcc 4 0 dc 15 * common collector amplifier: mid- and low-frequency analysis.model m1 npn bf=350.width in=80 out=80 * * dc bias circuit r1 2 4 r k vcc 4 0 dc 15 International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013 Automatically builds a Pegasus Workflow to manage these jobs

Copyright © 2012 HUBzero Foundation, LLC Submit: Parameter Sweeps 13 If you normally run a command like this… spice3 –b circuit Submit to remote resources like this… submit –v DiaGrid spice3 –b circuit submit –p spice3 submit –p –p spice3 submit –d indata.csv spice3 submit –p spice3 –b submit –p spice3 –b International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013

Copyright © 2012 HUBzero Foundation, LLC Submit: Parameter Sweeps International Workshop on Science Gateways, ETH Zürich, June 3-5, If you normally run a command like this… spice3 –b circuit Submit to remote resources like this… submit –v DiaGrid spice3 –b circuit submit –p spice3 submit –p –p spice3 submit –d indata.csv spice3 submit –p spice3 –b submit –p spice3 –b

Copyright © 2012 HUBzero Foundation, LLC Rappture Toolkit International Workshop on Science Gateways, ETH Zürich, June 3-5, Rappture Builder Your Code

Copyright © 2012 HUBzero Foundation, LLC Publishing Simulation Tools 16 Fingertip access to latest tools Tools run in the Cloud Leverage HPC resources Hub reports impact metrics Rappture International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013

Copyright © 2012 HUBzero Foundation, LLC Tools leveraging Pegasus International Workshop on Science Gateways, ETH Zürich, June 3-5, OpenSees Laboratory NEES.org BLASTer DiaGrid.org

Copyright © 2012 HUBzero Foundation, LLC Integration Challenges International Workshop on Science Gateways, ETH Zürich, June 3-5, Hub Web Server Execution Host 1 Execution Host 2...  Dynamic mapping to resources  Staging files in and out  Reporting job status  Reporting usage and metrics “submit” server head node Shared HPC Filesystem Condor IO GridFTP pegasus-status pegasus-statistics pegasus-analyzer pegasus-status pegasus-statistics pegasus-analyzer Pegasus 4.0

Copyright © 2012 HUBzero Foundation, LLC More information Michael McLennan Director, HUBzero® Project 19 Ewa Deelman Assistant Director Science Automation Technologies International Workshop on Science Gateways, ETH Zürich, June 3-5, 2013