1 Is there an ? Is there an app for that ? Challenges in scalable analysis for Life sciences 1 Nirav Merchant UA BioComputing + iPlant Arizona Research Laboratories University of Arizona

Topic Coverage  Formula for success (and failure)  Flavors of Bio-information  What is iPlant ?  Typical Non-NGS workflow  Data life cycle issues (some)  Application life cycle issues (some)  Why “app” ? 2

3 + = Simple Formula

The Reality 4 ++ PERL Python Java Ruby Fortran C C# C++ R Matlab etc. PERL Python Java Ruby Fortran C C# C++ R Matlab etc. Amazon Azure Rackspace Campus HPC XSEDE Etc. Amazon Azure Rackspace Campus HPC XSEDE Etc. and lots of glue…..

+ = Simple Formula

Life science: Going across scales 6

Putting it all to work Wayne Stayskal, The Tampa Tribune

The iPlant Collaborative Cyberinfrastructure for the Plant Sciences The iPlant CI is designed as infrastructure. This means it is a platform upon which other projects can build. Use of the iPlant infrastructure can take one of several forms: Storage Computation Hosting Web Services Scalability

 For a challenge as broad as “plant science,” focus on specific applications/tools is a moving target, and never enough.  Most important to build a platform that can support diverse and constantly evolving needs. “Cyberinfrastructure” is, in fact, infrastructure. The platform can lift all the apps, not select winners and losers. “ The useful lifetime of our analysis toolchains is now 6 months” -Matthew Trunnel, Broad Institute The iPlant Collaborative Cyberinfrastructure for the Plant Sciences

End Users Computational Users Teragrid XSEDE The iPlant Collaborative Cyberinfrastructure for the Plant Sciences

BioInformation :: Data Flavors  Sequences  Structures  Images  Video  Audio  Pathways (graphs)  Text (Publications)  Traces  Combination (eg Video & Traces)  And much more …

Life scientist :: Data Wrestler  Volume of data is increasing  Resolution of data is increasing  Number of data repositories is increasing  Ever increasing analysis options  Demands to share, collaborate data (team science)  Do you know where your data is ? (and your collaborators data !)

13 System s Biology Genomi cs Function al Genomic s Metabolomi cs Proteomi cs Pharmaco- genomics Modelin g Clinical Pathway s

X prize for sequencing guidelines are different, this is graphics dated

X prize for analyzing it ? ? 15

The Lifecycle 16 The Fourth Paradigm: Data-Intensive Scientific Discovery

17

18

Why is this hard when we have …  Pegasus  Taverna  Kepler  Condor (DAGman)  Gearman  Makeflow  myExperiment  Science pipes  We have X (take your pick) 19

What did the scientists do ? 20 Used the “parametric launcher” Essentially its a very functional “submit” script ! Why use it ? Dir of full of files and one executable Simple linear flow (no branching) Needed results “yesterday” for conference/working group Need to be run ONCE every year Not sexy but functional Serial runs are important

Python in HPC : OMG 21

Data issues 22

DLM: Issues  Most “pipelines/analysis” are Data intensive Sadly data originates from slow desktops, external hard drives, file servers using ftp, http etc (and ends up there)  Hard to stage data to begin computation ! No place to bring things together (quickly)  Data needs substantial pre and post processing Meta data is usually not adequate  RDBMS are part of workflows Do you need better indexing of flat files ?  It does not have to be this way ! 23

24

Data Lifecycle: Our effort 25

What can users do ? 26

27

But I don’t get throughput 28 Networking is huge BLACK BOX and too much finger pointing

Compute Issues: Cloud 29

What is cloud computing ?

The application lifecycle 31

 A rich web client  Provides a consistent interface to a range of bioinformatics tools  Provides a portal to users not wishing to interact with lower level infrastructure  An integrated, extensible system of applications and services  Provides additional intelligence above low level APIs – Provenance, Collaboration, etc. 32 The iPlant Collaborative iPlant Discovery Environment

 API-compatible implementation of Amazon EC2/S3 interfaces  Virtualize the execution environment for applications and services  Get Up to 12 core / 48 GB instances  Access to Cloud Storage + EBS  1008 users  167 users launched 657 instances (May 2012)  227 were terminated outside the of Atmosphere due to idleness (per user's request)  430 instances average time was 1 day, 16 hours, and 13 minutes. Longest running was 30 days  Run servers, CloudBurst desktop use cases. Big data and the desktop are co-local again! >60 hosted applications in Atmosphere today, including users from USDA, Forest Service, data providers, etc. 30+ private images for postdocs and grad students for training classes The iPlant Collaborative Project Atmosphere™: Custom Cloud Computing

Atmosphere: Collaboration iPlant Data Store

Lifecycle

How to Connect

Different Ways to Log in to VMs

Steps to get started !

My wish list for CCL (parrot)  Improved performance for iRODS transfers (parallel transfers ?)  File permission calls (iRODS ACL)*  Ability to provide throughput/transfer stats  Thanks for updating iRODS support to

My wish list for CCL (makeflow)  *Bundle dependencies along with script and binaries e.g. CDE: Automatically create portable Linux applications  Progress reporting, profiling of performance e.g equivalent progress bar 40 *Not a makeflow issue but a good feature

Staff: Greg Abram Sonali Aditya Roger Barthelson Brad Boyle Todd Bryan Gordon Burleigh John Cazes Mike Conway Karen Cranston Rion Doodey Andy Edmonds Dmitry Fedorov Michael Gatto Utkarsh Gaur Cornel Ghiban Michael Gonzales Hariolf Häfele Matthew Hanlon 74 MetadataDataToolsWorkflowsViz Executive Team: Steve Goff Dan Stanzione Faculty Advisors & Collaborators: Ali Akoglu Greg Andrews Kobus Barnard Sue Brown Thomas Brutnell Michael Donoghue Casey Dunn Brian Enquist Damian Gessler Ruth Grene John Hartman Matthew Hudson Dan Kliebenstein Jim Leebens-Mack David Lowenthal Robert Martienssen Students: Peter Bailey Jeremy Beaulieu Devi Bhattacharya Storme Briscoe Ya-Di Chen John Donoghue Steven Gregory Yekatarina Khartianova Monica Lent Amgad Madkour B.S. Manjunath Nirav Merchant David Neale Brian O’Meara Sudha Ram David Salt Mark Schildhauer Doug Soltis Pam Soltis Edgar Spalding Alexis Stamatakis Ann Stapleton Lincoln Stein Val Tannen Todd Vision Doreen Ware Steve Welch Mark Westneat Andrew Lenards Zhenyuan Lu Eric Lyons Naim Matasci Sheldon McKay Robert McLay Angel Mercer Dave Micklos Nathan Miller Steve Mock Martha Narro Praveen Nuthulapati Shannon Oliver Shiran Pasternak William Peil Titus Purdin J.A. Raygoza Garay Dennis Roberts Jerry Schneider Anthony Heath Barbara Heath Matthew Helmke Natalie Henriques Uwe Hilgert Nicole Hopkins Eun-Sook Jeong Logan Johnson Chris Jordan B.D. Kim Kathleen Kennedy Mohammed Khalfan Seung-jin Kim Lars Koersterk Sangeeta Kuchimanchi Kristian Kvilekval Aruna Lakshmanan Sue Lauter Tina Lee Bruce Schumaker Sriramu Singaram Edwin Skidmore Brandon Smith Mary Margaret Sprinkle Sriram Srinivasan Josh Stein Lisa Stillwell Kris Urie Peter Van Buren Hans Vasquez-Gross Matthew Vaughn Fusheng Wei Jason Williams John Wregglesworth Weijia Xu Jill Yarmchuk Aniruddha Marathe Kurt Michaels Dhanesh Prasad Andrew Predoehl Jose Salcedo Shalini Sasidharan Gregory Striemer Jason Vandeventer Kuan Yang Postdocs: Barbara Banbury Jamie Estill Bindu Joseph Christos Noutsos Brad Ruhfel Stephen A. Smith Chunlao Tang Lin Wang Liya Wang Norman Wickett The iPlant Collaborative