Using the ARCS Grid and Compute Cloud Jim McGovern

QuestNet 2011 First thing – get quota in the ARCS Compute Cloud. By registering with ARCS Access Service, you will automatically be given quota in the ARCs Compute cloud. Some Housekeeping

QuestNet 2011 Compute Resources used by Researchers Researchers may access a range of resources to meet their compute needs. For example Their own PC Their own University servers or HPC A regional HPC facility, e.g. QCIF Facilities at another university (possibly national) remote merit-based allocation reserved for their discipline or project. E.g. NCI, VLSCI, QFAB.

QuestNet 2011 National HPC Resources

QuestNet 2011 Problem: Multiple Systems More places to submit jobs Even more to learn (submission systems, quirks, etc) Need accounts set up at each site Firewall dramas, etc

QuestNet 2011 Problems associated with Access to Diverse, Remote Compute Resources Different submission systems Different accounts and authentication Different administration systems/ people to deal with. Different file systems and difficulty in sharing output or using shared input. Need for support/ capability

QuestNet 2011 Solution: Grid - Unified Submission System Use Globus One submission system to understand Handles local quirks automagically A kind of single login (via certificates) However...

QuestNet 2011 Example Job Script /usr/bin /whoami ${GLOB US_SCRATCH_DIR } ${GLOBU S_SCRATCH_DIR} /stdout.samtest ${GLOBUS _SCRATCH_DIR}/s tderr.samtest 60 single gsiftp:// ngdata.ivec.org:2811 /etc/termcap file: ///${GLOBUS_SCR ATCH_DIR}/file.sa mtest file:///${ GLOBUS_SCRATC H_DIR}/file.samtest gsif tp://ng2.vpac.org:281 1/${GLOBUS_USE R_HOME}/file.samt est22 file:///${GLO BUS_SCRATCH_DI R}/file.samtest

QuestNet 2011 Better Solution: Grisu Single Grisu core with access to all resources Client machine connects to this core to manage all aspects of job creation, submission, and result collection Provides a simplified API for the Grid Reduces firewall issues Shibboleth logins

QuestNet 2011 Even Better Solution: The ARCS Compute Cloud Access via applications (“application as service”) One account gives access to all Shibboleth login Minimal input with standardised implementations App 1App 2App n

QuestNet 2011 ARCS Compute Cloud Provides a web-based Graphical User Interface for job submission Same interface for all systems, even though target is different. Uses AAF for authentication – use your own institution's credentials Is integrated with ARCS Data Fabric Provides support resources

QuestNet 2011 ARCS Compute Cloud Provides useful resources across many HPC resources that any one can access. Provides Application as a Service: access package, hides Compute Resource Selects best location to run jobs from those available. Minimises user input Can free up desk top (e.g. rendering, statistical analysis)

QuestNet 2011 Demonstration: Running an application through ARCS Compute Cloud

QuestNet 2011 Single Web-based GUI for HPC job management Input from Data Fabric/Desk top Output to Data Fabric Login with your home institution's credentials Ability to use defaults only. Re-cap on demonstration

QuestNet 2011 Go to ARCS Web site and select the “Services Login” Login using your IdP Select Compute Cloud Select New Job Select Underworld application from list on LHS Select Data Fabric for input files. First underworld file is primary input file, and second is additional input file. Submit Job Monitor your job Review output files Run an application through ARCS Compute Cloud

QuestNet 2011 Questions?

QuestNet 2011 Extending the ARCS Grid and Compute Cloud Jim McGovern

QuestNet 2011 The ARCS Compute Cloud provides a useful facility for those who want to use a standard available application in a standard way. The ARCS facilities also support: Roll your own code New applications (new templates) Restricted resources or applications Building your own workflows: Using Java Using Scripting languages (e.g. Python) Accessing other resources on the ARCS Grid

QuestNet 2011 Roll your own code: - Java template - Python Scripts - Custom Scripts Accessing other resources on the ARCS Grid

QuestNet 2011 What resources are available on the Grid? See This provides all information, links to HPC providers. Accessing other resources on the ARCS Grid

QuestNet 2011 Building new templates Create a text file of properties Submit to ARCS Place in a particular repository folder – e.g. under Earth Sciences with.template extension Automatically put in Web interface (Compute Cloud) Produces RSL that is translated to local submission system Accessing other resources on the ARCS Grid

QuestNet Accessing other resources on the ARCS Grid

QuestNet 2011 commandline = Underworld ${file} application = Underworld applicationVersion = = Generic = - [file] type = SingleInputFile title = Primary XML Input File size = 2000x70 filter = BasenameFilter - [file2] type = MultipleInputFiles title = Additional Files size = 2000x210 filter = BasenameFilter Building New Templates

QuestNet 2011 [cpus] type = Cpus title = CPUs size = 100x100 DefaultValue = 4 - [walltime] type = Walltime title = Walltime defaultAmount = 10 defaultUnit = hours size = 200x100 #[ ] #type = #size = 400x100 Building New Templates - [jobname] type = Jobname defaultValue = underworld title = Jobname size = 2000x70

QuestNet 2011 Restricted Applications and Facilities: – Licensed software – Restricted allocations Merit allocations Accessing other resources on the ARCS Grid

QuestNet 2011 Create a VO that embodies particular accounts. (approved by sys admins) Map shib credentials to those VOs. E.g. – QCIF users – Fluent users – NCI/QFAB/VLSCI users Accessing other resources on the ARCS Grid

QuestNet 2011 Supporting your own work flows/ interfaces: – BES Standard interface Web services that allow management of activities including processes that run compute jobs – RESTful interface Easier to invoke web services suitable for quick easy development with Python clients for example – SLCS Delegation Service Use Shib credentials in your apps Accessing other resources on the ARCS Grid

QuestNet 2011 Adding your own resources: – Add your facility/ resource (Condor) to Grid – Documentation – Assistance – Set up and test Globus gateways – Create MDS (Information Service) entries Accessing other resources on the ARCS Grid

QuestNet 2011 The compute cloud can be tailored for: compile and run apps new templates particular sets of users (licensees/ authorised users) particular compute resources Specific work flows and interfaces Add your resource to grid Non-Standard use Summary

QuestNet 2011 Standard Applications: Register with ARCS and you will have access to these. For other applications, projects or work flows, call the ARCS help Desk and we will work with you to set up what is needed. How to get started?

QuestNet 2011 Questions?