iPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Collaborating with iPlant

iPlant is not a company! We’re colleagues We’re potential collaborators

Collaborating with iPlant Image from: Our goal is solve computational bottlenecks that impede research. Bottlenecks might be storage space, but it also might be finding a way to make tools easier to use. Doing this properly requires community input!

Collaborating with iPlant iPlant is Infrastructure. Like most infrastructure, you don’t always use it directly as an end user. Often, it is hidden underneath some other service.

Collaborating with iPlant Other major projects are beginning to adopt the iPlant CI as their underlying infrastructure (some completely, some in limited ways): – CoGe (auth service, hosting) – BioExtract (web service platform) – CiPRES (computation) – Gates Integrated Breeding Platform (hosting, development)

Collaborating with iPlant API The API (Application Programmer Interface) is the way bioinformatics tools and data get integrated with iPlant. The API is a straightforward web service. – You can use it with a simple curl command, e.g.: Paste example No different than embedding a Google API or map. You can use this for just authentication, for transferring files, running computation, looking up taxonomic names, etc.

Collaborating with iPlant API exemplar users Carol Lushbough, Bioextract BioExtract, a DBI funded portal, is currently being re- written to take advantage of the API. Carol is running jobs on Lonestar through the API. The API has also been adopted outside plants, by Apache Airavata, CyberGIS, and some XSEDE sites.

Collaborating with iPlant Stampede - High Level Overview Base Cluster (Dell/Intel/Mellanox): – Intel Sandy Bridge processors – Dell dual-socket nodes w/32GB RAM (2GB/core) – 6,400 nodes – 56 Gb/s Mellanox FDR InfiniBand interconnect – More than 100,000 cores, 2.2 PF peak performance Co-Processors: – Intel Xeon Phi “MIC” Many Integrated Core processors – Special release of “Knight’s Corner” (61 cores) – All MIC cards are on site at TACC more than 6000 installed final installation ongoing for formal summer acceptance – 7+ PF peak performance Max Total Concurrency: – exceeds 500,000 cores – 1.8M threads Entered production operations on January 7, 2013

iPlant is the first project to make most of the XSEDE resources available via a simple web service interface. iPlant can help you deploy an application on these resources. With a few lines of code, your program or web site can directly utilize some of the world’s largest systems, without you or your users ever logging in. Collaborating with iPlant API

Collaborating with iPlant Data Store iPlant now stores hundreds of millions of files, with nearly a petabyte of data under management. If you need to store or distribute a dataset, you can use the data store as your mechanism. The API is one way to do this, but there are also non-web methods for bulk data transfer. Once in the data store, a variety of interfaces can be used to access/retrieve the data – Indexing and analytics capabilities coming soon.

The current version supports multiple sites, but only full mirrors (all data at every site). The next version will allow new storage sites to enroll, but only keep data of interest. – With this option, iPlant could act as a mirror for specific projects/databases without that project hosting “unrelated” data. – Relevant data could be cached at local sites for faster access. – With a query capability, an internationally federated data store could be built where *no* site has all the data, but all sites can find it, allowing greater scaling. Collaborating with iPlant Data Store

Atmosphere is now offering (in beta) a “Metal as-a-service” mode. Servers running Linux are set up at a site, we install the Atmosphere back end. From the main Atmosphere site, you can route your VMs back to your local hardware. – Increase allocation – Control images and data locally. Collaborating with iPlant Atmosphere

If you *don’t* have your own web application, but you have a program, workflow, or dataset you would like to widely distributed, iPlant can be the platform. – Use the DE to share your workflow or program ( working on direct links that can be provided to use in publications or from your website). – Use the data stores public space to store and distribute your data (replicated, many interfaces) Collaborating with iPlant Distributing your science

CI as “Everyday” Science

iPlant Advanced Collaborative Support Provide a computing expert for an extended period of time to rebuild a popular tool for scalability, or other key functionality – Could be scaling, info/visualizations, or just information architecture help. – A typical engagement may run a month to 6 months. – An expert focused on making *your* particular tool or workflow run well on our systems. – Deal with the complexity of coming large scale systems (millions of threads).

Shaping iPlant to Suit Your Science Your Feedback is Critical iplantc.org/tswpost