1. ACI-REF Barr von Oehsen, Clemson University OSG ALL HANDS MEETING March 25, 2015

2. Background & Context

3. A Novel Approach Research & Education “Facilitators” (REFs) Domain-area experts with knowledge of ACI resources and capabilities Substantial outreach activities by REFs – to all disciplines and departments on campus One-on-one problem solving and interaction Focus on people helping users with technology Notion of a “research concierge” Sharing expertise across institutions ACI Resources Researchers ACI-REF Project Facilitators

4. Formation NSF-sponsored workshops held in 2012 helped define the needs of the broader community Goal: Advance our nation's research & scholarly achievements through the transformation of campus computational capabilities and enhanced coupling to the national infrastructure.

5. Reference Point - Scale

6. A Model – Campus Level Clemson User Growth May 2010 – first Clemson “facilitator” funded 2008: 19/52 Departments Trained on HPC 2014: 46/52 Departments Trained on HPC May 2010: NSF Outreach & Infrastructure Improvement Grant Funded Clemson ACI-REF hire preparing to serve the GIS communities

7. Business growing: some real unusual suspects Department Bioengineering CCIT HPC CCIT Software Development Chemical Engineering Chemistry Civil Engineering Customer Rel & Learning Tech Cyberinfrastructure Tech Integ Economics Elec. & Computer Engr. Environmental Engr & Earth Sci Experiential Education General Engineering Genetics & Biochemistry Industrial Engineering Infrastructure Services & Ops Department International Programs Law Enforcement & Safety Management Mathematical Sciences Mechanical Engineering Medicaid IT Services Network Services & Telecomm Physics And Astronomy Public Health Sciences Research Safety School of Ag for Env Science School of Computing Univ Facilities Support Svcs VP Finance & Operations 7 Hadoop Training Classes

8. Activities & Progress To Date

9. Progress Bi-weekly facilitator meetings focused on collaborative projects, sharing of advice & expertise, and discussions on opportunities Facilitating a community for NSF-funded CI-Engineers through Chris Konger (Clemson), funded through the project to bridge technical and research areas and explore new workflows for enabling scientists. University of Missouri’s and the University of Oklahoma’s CC*IIE CI-Engineer projects were funded (the ACI-REF project supplied a letter of collaboration) and work will begin soon.

10. Progress Sharing user support requests between campuses Clemson & Utah facilitators working together on computational chemistry support issues, Harvard addressing bioinformatics inquiries from Clemson researchers University of Southern California needed help setting up an Intel Xeon PHI node on their cluster. Question was submitted to facilitator email list. University of Wisconsin, Harvard, and Clemson responded with configuration suggestions as well as sharing contact information with experts at TACC. Utah and USC facilitators have accounts on the Clemson cluster

11. Progress Sharing user support requests between campuses Wisconsin user was having issues running LAMMPS from an MPI- implemented python program. Galen Collier (Clemson) and Wim Cardoen (Utah) worked with Lauren Michael (Wisconsin) to address the issue. The user's issue was resolved after several tips from Galen and Wim. Clemson is working with ACI-REF schools on hosting regional training sessions on the Intel Xeon PHI. New GIS focused training is being developed and will be available to all ACI-REFs. Several schools have adopted office hours – based on Harvard model

12. Progress Coordinated OSG training: Emelie Harstad, Chander Sehgal, Balamurugan Desinghu Clemson was the test case Followed with tutorial for all of the ACI-REFs Very positive feedback Next steps with OSG In discussion with Rob Gardner about coupling Clemson’s Palmetto cluster with OSG Interested in being able to offload high throughput jobs directly from Palmetto to OSG (without the user necessarily knowing) If we can successfully do this, many of the ACI-REF schools would be doing this Would be very interested in including Docker/Containers into this environment.

13. Year 1 Successes - Wisconsin Complex Economic Modeling – Nicolas Roys, Jesse Gregory, and Amit Gandhi, University of Wisconsin-Madison A number of researchers in the Department of Economics at UW-Madison have benefited from the assistance of ACI-REFs in designing high-throughput computational methods for solving complex economic models that are otherwise avoided by economists all over the world for their dependence on vast amount of computational time. As a result of consulting with ACI-REFs to optimize the computational approach, campus economists -- including Nicolas Roys, Jesse Gregory, Amit Gandhi, and students they advise -- can achieve up to decades of computing in a single day by simultaneously leveraging campus compute capacity and that of the Open Science Grid. Example: http://www.opensciencegrid.org/using-high-throughput- computing-to-evaluate-post-katrina-rebuilding-grants/ http://www.opensciencegrid.org/using-high-throughput- computing-to-evaluate-post-katrina-rebuilding-grants/

14. Year 1 Successes - Utah High-Energy Theoretical Physics – Chris Kelso, University of Utah “I work in high energy theoretical particle physics. Specifically, I investigate physics beyond the Standard Model with a focus on dark matter implications. My research often requires scans of models that have very large numbers of parameters. This work could not be completed without the computing resources provided at CHPC. Almost as valuable as the use of the CHPC machines was the extremely helpful assistance I received from Wim R. Cardoen. Many of the codes I often use are serial, open source code that has been developed by many physics experts. To try and convert these codes to parallel would be a monumental task. Wim worked very hard to help me to find a solution that allowed this serial code to still utilize the numerous processors available on the CHPC machines. Without this, my projects would take months to finish, rather than a few days.” – Chris Kelso, University of Utah PostDoc, on Utah ACI-REF Wim Cardoen

15. Year 1 Successes - Harvard HPC Assistance in Biology Software and Workflow – Zack Lewis, Harvard University “I am a sixth year graduate student in the Department of Organismic and Evolutionary Biology. I started a transcriptomics project with little experience in coding and no experience in high powered computing (HPC). Without Bob Freeman’s work through ACI-REF I do not think I would have been able to complete my bioinformatics project. I was not aware of ACI- REF at the time I started my HPC bioinformatics work. To my good fortune I happened to connect with Bob Freeman at the weekly Research Computing office hours. Bob has accompanied me nearly every step of the way along my 6 month journey into HPC. Bob’s help has taken the form of instruction on coding, monitoring active jobs, writing and adapting scripts for my project, as well as connecting me with researchers working on similar problems or at similar stages in learning transcriptomics. In particular, building connections with other researchers at Harvard through ACI-REF has been one of the most useful experiences. I now often work through my HPC issues with graduate student and postdoc peers that I have connected with through Bob.” – Zack Lewis, Harvard University PhD Candidate, on Harvard ACI-REF Bob Freeman

16. Year 1 Successes - Clemson Optimizing Genomics Database Transfers – Alex Feltus, Clemson Genomics research is one of the largest drivers in generating Big Data for science, with the potential to equal if not surpass the data output of the particle physics community. Like physicists, university-based life-science researchers must collaborate with counterparts and access data repositories across the nation and globe. Dr Feltus’ collaboration with facilitators and network engineers has been an imperative to produce meaningful and significant outcomes. Leveraging Internet2 community network and human connections, along with key NSF funding, the collaborative efforts of Clemson and NLM have essentially changed Feltus’ workflow to achieve scientific results at a larger scale and with massive gains in data mobility across networks. “The significance of the speed up (which is looking more like 75-100X by the way) is that I can: A) SCALE UP EXPERIMENTS by using more input data since I can get the data quickly; and B) MINIMIZE LOCAL STORAGE of huge files because they enter workflows and then get deleted. I can just download them again if I screwed up my experiment.“ -Alex Feltus, Clemson University Faculty

17. Progress Example: Feltus Genomics Project UH UA = ACI-REF School = Tripal Genome DB = GENI Rack = AL2S The Feltus lab @Clemson is interested in optimizing genomics data transfer between Tripal+ genome database sites with Internet2/GENI and SDN. Utah now duplicating. 27.8X Faster Transfer from NCBI to Clemson Cluster! The significance of the speed up (which is looking more like 75-100X by the way) is that I can... A) SCALE UP EXPERIMENTS by using more input data since I can get the data quickly. B) MINIMIZE LOCAL STORAGE of huge files because they enter workflows and then get deleted. I can just download them again if I screwed up my experiment. -Alex Feltus, Associate Professor – Genomics (Clemson University)

18. Challenges Members of this community are used to working on their own It's hard to break habits and easy to fall back into routines Working within a tight time frame Recognizing when a steering correction needs to be made Metrics How do we know we are successful? First goal: Enable scientific and research outcomes that would not be possible without the project and ACI-REF community. Standard Reporting Mechanisms

19. Challenges Building research collaborations across sites Having faculty/research advocates help (for example, Alex Feltus) Using an NSF solicitation as catalyst: CloudLab Getting the word out on our campuses as to how ACI-REF can help a researcher be more successful Onboarding New Facilitators Communication Technology stacks vary More ideas than time to execute

20. Beyond the ACI-REF Project: Forming a National Consortium on Research Facilitation

21. Moving Forward Sustainability strategies for (a) the project and (b) for a larger group – the ACI-REF Consortium Mechanism for bringing on new campuses committed to a community that: Values facilitation as a critical need to support research at a campus & national level Focuses on people helping people use the technology – and invests in their professional development Collaboration - sharing of expertise and knowledge across campuses Sustainability - creation & adoption of a new career path for facilitators Effort couples with existing campus & national investments so as to maximize impact on existing and planned resources

22. Remaining Questions Timeline & Roll-out of Consortium Activities Fiscal Agent (for dues collection) Consortium Staffing & Organization Finalize Governance Plan (Bylaws) Structuring approach to Continuing NSF (and other) support during the consortium “stand-up” phase Inclusion of a formalized technical track to help communities improve their workflow based on early work done in Genomics (Alex Feltus) Creation of “business intelligence” tools to help convey the value added message

23. Consortium Organization The [ACI-REF] Council is the primary oversight body for the activities of the [ACI-REF] Consortium and determines the strategic direction of the Consortium. Members will be entitled to a seat on the Council. The [ACI-REF] Council Chair is elected by the Council to serve a two- year term. A Chair can serve no more than two full, consecutive terms. The Executive Director (title TBD) is selected by the Council to manage programmatic and outreach activities with the assistance of the Executive Team. The Executive Director shall be eligible for re-appointment every two years by the Council. The Executive Team is appointed by the Executive Director with approval from the Council to direct certain aspects of the Consortium as delegated by the Executive Director. The Executive Team drafts policies for Council approval and represents the [ACI-REF] Consortium in meetings with other organizations and initiatives. The Executive Team, initially, may be a group of volunteers from the member campuses.

24. Consortium Organization