A Swift Talk about Globus Technology: What Can It Do for Me? OOI Cyberinfrastructure Design Meeting, San Diego, October The Globus Team (presented by Kate Keahey)

10/18/07, ORION meetingVirtual Workspaces: Overview l Globus Toolkit overview l What is WSRF and how does it work? l Defining community: membership and laws l Workflow management with Swift l Globus as project: how we work

10/18/07, ORION meetingVirtual Workspaces: What Is the Globus Toolkit? l The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications l Heterogeneity u To date (v1.0 - v4.0), the Toolkit has focused on simplifying heterogeneity for application developers u We are increasingly including more “vertical solutions” that implement typical application patterns l Security u The Grid Security Infrastructure (GSI) allows collaborators to share resources without blind trust l Standards u Our goal has been to capitalize on and encourage use of existing standards (IETF, W3C, OASIS, GGF) u The Toolkit also includes reference implementations of new/proposed standards in these organizations

10/18/07, ORION meetingVirtual Workspaces: What’s In the Globus Toolkit? l A Grid development environment u Develop new OGSA-compliant Web Services u Develop applications using Java or C/C++ Grid APIs u Secure applications using basic security mechanisms l A set of basic Grid services u Job submission/management u File transfer (individual, queued) u Database access u Data management (replication, metadata) u Monitoring/Indexing system information l Tools and Examples l The prerequisites for many Grid community tools

10/18/07, ORION meetingVirtual Workspaces: Globus Technology Areas l Core runtime u Infrastructure for building new services u WSRF = Web Service Resource Framework l Security u Apply uniform policy across distinct systems l Execution management u Provision, deploy, & manage services l Data management u Discover, transfer, & access large data l Monitoring u Discover & monitor dynamic services

10/18/07, ORION meetingVirtual Workspaces: Incubator Projects Security Execution Mgmt Info Services Common Runtime Globus Projects Other MPICH G2 GridWay Data Mgmt Incubation Mgmt Cog WF LRMA GAARDS OGROGDTEUGP HOC-SAPURSE GridShib Introduce Dyn Acct WEEP Gavia JSC Gavia MS DDM Virt WkSp SGGC Metrics ServMark GridFTP Reliable File Transfer OGSA-DAI GRAM MDS4 CAS Data Rep Delegation Replica Location Java Runtime C Runtime Python Runtime C SecGT4 Docs MEDICUS GSI- OpenSSH MyProxy SwiftMonMan NetLogger GEMLCA Globus Toolkit

Web Services Resource Framework (WSRF)

10/18/07, ORION meetingVirtual Workspaces: “Stateless” vs. “Stateful” Services l Without state, how does client: u Determine what happened (success/failure)? u Find out how many files completed? u Receive updates when interesting events arise? u Terminate a request? l Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state Client FileTransfer Service move (A to B) move

10/18/07, ORION meetingVirtual Workspaces: FileTransferService (without WSRF) l Developer reinvents wheel for each new service u Custom management and identification of state: transferID u Custom operations to inspect state synchronously (whatHappen) and asynchronously (tellMeWhen) u Custom lifetime operation (cancel) Client FileTransfer Service move (A to B) : transferID move state whatHappen tellMeWhen cancel

10/18/07, ORION meetingVirtual Workspaces: WSRF in a Nutshell l Service l State representation u Resource u Resource Property l State identification u Endpoint Reference l State Interfaces u GetRP, QueryRPs, GetMultipleRPs, SetRP l Lifetime Interfaces u SetTerminationTime u ImmediateDestruction l Notification Interfaces u Subscribe u Notify l ServiceGroups RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR

10/18/07, ORION meetingVirtual Workspaces: FileTransferService (w/ WSRF) l Developer specifies custom method to createResource and leaves the rest to WSRF standards: u State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR) u State inspected by standard interfaces (GetRP, QueryRPs) u Lifetime management by standard interfaces (Destroy) Client FileTransferService createResource (A to B) : EPR createResource RPs Transfer getRP queryRPs destroy

10/18/07, ORION meetingVirtual Workspaces: The Introduce Authoring Tool l Define service l Create skeleton l Discover types l Add operations l Configure security l Modify service Introduce: Hastings, Saltz, et al., Ohio State University Generates GT4- compatible Web Services

10/18/07, ORION meetingVirtual Workspaces: Defining Community: Membership and Laws l Identify VO participants and roles u For people and services l Specify and control actions of members u Empower members  delegation u Enforce restrictions  federate policy A 12 B 12 A B Access granted by community to user Site admission- control policies Effective Access Policy of site to community

Courtsey : DOE report : LBNL : Authorization & Attribute Certificates for Widely Distributed Access Control XACML SAML

10/18/07, ORION meetingVirtual Workspaces: Globus Authorization Framework VOMSShibbolethLDAP PERMIS … GT4 Client GT4 Server PDP Attributes Authorization Decision PIP

10/18/07, ORION meetingVirtual Workspaces: Swift System l Clean separation of logical/physical concerns u XDTM specification of logical data structures + Concise specification of parallel programs u SwiftScript, with iteration, etc. + Efficient execution on distributed resources u Karajan threading, Falkon provisioning, Globus interfaces, pipelining, load balancing + Rigorous provenance tracking and query u Virtual data schema & automated recording  Improved usability and productivity u Demonstrated in numerous applications

10/18/07, ORION meetingVirtual Workspaces: Workflow Language - SwiftScript l Goal: Natural feel to expressing distributed applications u Variables (basic, data structures) u Conditional operators (if, foreach, ) u Functions (atomic / compound) l Used to connect outputs to inputs l It does not specify invocation order, only dependencies l It can be seen as a metadata for expressing experiments

10/18/07, ORION meetingVirtual Workspaces: Execution Engine l Karajan engine (event-based execution) l Has a scheduler to map tasks to resources u Score-based planning u Recovers from failures (retries) l Falkon resource manager creates a “private cluster” u Uses Globus GRAM4 (PBS/Condor/Fork) to acquire resources from Grid systems

Virtual Node(s) SwiftScript Abstract computation Virtual Data Catalog SwiftScript Compiler SpecificationExecution Virtual Node(s) Provenance data Provenance data Provenance collector launcher file1 file2 file3 App F1 App F2 Scheduling Execution Engine (Karajan w/ Swift Runtime) Swift runtime callouts C CCC Status reporting Provisioning Resource Provisioners Falkon/VWS Amazon EC2 Dynamic Provisioning: Swift Architecture Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford

10/18/07, ORION meetingVirtual Workspaces: Globus Philosophy l Globus was first established as an open source project in 1996 l The Globus Toolkit is open source to: u Allow for inspection l for consideration in standardization processes u Encourage adoption l in pursuit of ubiquity and interoperability u Encourage contributions l harness the expertise of the community l The Globus Toolkit is distributed under the (BSD-style) Apache License version 2

10/18/07, ORION meetingVirtual Workspaces: dev.globus l Governance model based on Apache Jakarta u Consensus based decision making l Globus software is organized as several dozen “Globus Projects” u Each project has its own “Committers” responsible for their products u Cross-project coordination through shared interactions and committers meetings l A “Globus Management Committee” u Overall guidance and conflict resolution

10/18/07, ORION meetingVirtual Workspaces: Guidelines (Apache Jakarta) Infrastructure (CVS, , bugzilla, Wiki) Projects Include …

10/18/07, ORION meetingVirtual Workspaces: Open Source != “Free time” l Globus development is well-funded u The open source model facilitates contributions u NSF and DOE sponsor Globus development at several institutions via multiple grants, totaling >$5M/yr u Non-U.S. science agencies also contribute to Globus development u Corporations also sponsor developers l NSF explicitly funds Globus improvements u CDIGS: Community-Driven Improvements to Globus Software

10/18/07, ORION meetingVirtual Workspaces: Parting Thoughts l There is a wealth of available Grid technology that can be leveraged for this project u Security mechanisms, workflow, and other services l The Globus Toolkit is funded by NSF to support scientific communities.