1. Genesis II Open Source, OGSA Implementation Genesis II: Easy-to-use, Standards Based Grid Middleware Andrew Grimshaw Genesis II Team University of Virginia Open Source, OGSA Implementation

2. Genesis II Open Source, OGSA Implementation Outline Background Standards Genesis II Demo Conclusion

3. Genesis II Open Source, OGSA Implementation Background Specifications cannot exist in vacuum –Implementations vet Specifications –Implementation experience shows how specs interact Grids have been around for a while, but adoption remains low – why? –Usability –Gap between grid designers and grid users

4. Genesis II Open Source, OGSA Implementation Why have Grids failed to cross the chasm? Applications programmers are exposed to the complexity of the underlying environment –Failures, data/application management, logging, … –Only the enthusiasts want to live on the bleeding edge –Joe six-pack (biologist | chemist | economist | *) just wants to do their domain research – not do heavy duty hacking –Activation energy is too high Once they have spent the effort to port there is little leverage – applications written for one software stack will not work on other stacks – often they will not even work on different versions of the same software stack –They must suffer the pain all over again Business needs solutions not technology –HTC solutions (e.g., SGE, LSF, Condor) are easy to use – but not really needed cross-site. –Data sharing solutions are primitive and do not integrate well with the existing infrastructure

5. Genesis II Open Source, OGSA Implementation Genesis II Users FIRST! –Design the system from the ground up with the overriding mantra that users come first! –Users dont want to know about grids –Provide a secure, cohesive system in a production system available to users today! Provide an open source, reference implementation of the OGSA and OGSA-related specifications –Use standards and proto-standards available from the OGF and OGSA to –Provide feedback into the OGF process on various standards based on implementation experience

6. Genesis II Open Source, OGSA Implementation Outline Background Standards Genesis II Demo Next Steps

7. Genesis II Open Source, OGSA Implementation Standards Represented in Genesis II WSDL SOAP WS-Addressing WS-Security XACML SAML Shibboleth OGF OGSA WS-Naming BES ByteIO RNS JSDL

8. Genesis II Open Source, OGSA Implementation RNS Basic directory service maps strings to EPRs. –Similar to Unix directories (EPR instead of inode) –Can point to anything that has an EPR.

9. Genesis II Open Source, OGSA Implementation Pathnames traverse the graph /usr/bin/gcc / bin usrhosts BCA gcc RNS Directory Resource

10. Genesis II Open Source, OGSA Implementation OGSA-ByteIO

11. Genesis II Open Source, OGSA Implementation ByteIO Properties

12. Genesis II Open Source, OGSA Implementation JSDL – Job Submission Description Langauge XML based - extensible Widely adopted Five sections –Job structure –Job Identity –Job application –Job resource –Job data staging Not intended for human consumption

13. Genesis II Open Source, OGSA Implementation Example JSDL gnomad ribo ribo.d overwrite true rns:/home/mmm2a/data/ribo.567.x tmapfinal overwrite true rns:/home/mmm2a/res/final.567

14. Genesis II Open Source, OGSA Implementation OGSA-BES Basic Execution Service Service interface for starting and managing remote compute jobs JSDL as the job specification language Implementation is not specified –Queue –Fork/exec –Virtualize –Etc. –Emphasis on Basic Comes together in the HPC-BP

15. Genesis II Open Source, OGSA Implementation WSI-BSP – Basic Security Profile Most widely used WS security profile. Specifies many things, including how and where identity tokens are passed, their schema, and what they should be called. We support username/password, X.509, and SAML.

16. Genesis II Open Source, OGSA Implementation WS-SecureAddressing A profile on WS-Addressing and WS- SecurityPolicy Embeds specified URIs in EPR metadata that correspond to profiled security policies –Tells client the security policy of the refereced endpoint Embeds public key into EPR so that the client can communicate securely with the endpoint.

17. Genesis II Open Source, OGSA Implementation WS-Naming Profile on WS-Addressing EPR Two components –Endpoint Identifiers –Endpoint Resolution Dynamic rebinding to support traditional distributed system transparencies: migration, failure, replication, etc.

18. Genesis II Open Source, OGSA Implementation Outline Background Standards Genesis II Demo Conclusion

19. Genesis II Open Source, OGSA Implementation Goals Provide an easy-to-use production Grid system with compute and data Grid capabilities to various partner groups and research projects, Provide a fully functional Grid framework on which further Grid research could be performed, To test drive the various specifications making their way through various standardization organizations to both vet and better understand those specifications, both in isolation, and together as a whole.

20. Genesis II Open Source, OGSA Implementation Genesis II Features Ease-of-use – focus on user experience Standards based – both WS and OGF Data and compute grid Open Source – Java/Apache Axis Robust Secure

21. Genesis II Open Source, OGSA Implementation Users First A large percentage of a grids target audience is unable or unwilling to learn new interaction abstractions Instead of asking the user to adapt to the grid, we should adapt the grid to the user

22. Genesis II Open Source, OGSA Implementation User Abstractions One of the most ubiquitous user interaction abstractions is the file system –Drag-and-drop –Double Click –/proc filesystem –Named pipes RNS, ByteIO, and WS-Naming provide the foundation for building these abstractions

23. Genesis II Open Source, OGSA Implementation (Most) everything is a file or directory Files and directories can be accessed without knowing anything about Grids, Web Services, or anything thanks to FUSE/IFS –FUSE/IFS map the Grid into the file system BES resources, queues are directories –ls to list the jobs, cat a job to see its state –cp a JSDL file into the directory -> start the job up –A shell script can start jobs by copying Genesis II containers are directories –ls to see the services and porttypes IDP are files/directories ISs are directories –cp a query file to IS, creates result RDBMSs will be directories The user can access all of these services without dealing with Web Services!!

24. Genesis II Open Source, OGSA Implementation Genesis II - Data Grid shell –Usual commands, ls, cat, etc. FTPd for Windows FUSE for Linux IFS for Windows ExportDir tool Replicated ExportDir

25. Genesis II Open Source, OGSA Implementation Why FUSE? FUSE is a per-user proxy to the Grid –Permissions and access based on the users credentials FUSE maps Grid name space (RNS) into local operating system. The Grid looks like a local file system.

26. Genesis II Open Source, OGSA Implementation Genesis II - Compute Grid shell –Run, qsub, qdel, BES resources mapped into the file system –All BES resources implement HPC-BP and HPC-FSE Genesis queues are workload managers Both BES containers and queues support the RNS interface Applications can be run by copying JSDL files into containers JSDL tool Application Management

27. Genesis II Open Source, OGSA Implementation Genesis II - Security WSI-BSP –SSL/TLS and identity token(s) in SOAP header Authentication via –X.509 based Uses existing credential infrastructure –Signed SAML assertions (groups realized this way) –Username/token –login consists of specifying cert file or authenticating to a WS- Trust STS Message integrity via both transport and message level encryption and signing (configurable) Authorization via plug-able module –Currently using access control lists

28. Genesis II Open Source, OGSA Implementation Demonstration ssh onto Linux machine back home Windows interface from this machine in Catania Connected to XCG – Cross Campus Grid

29. Genesis II Open Source, OGSA Implementation Steps Grid shell –Ls /, /bes-containers, /groups, –Login & cd /home/grimshaw –ls, ls –epr, –Fuse mount In Linux, ls grid, cd home/grimshaw –Ls gnomad, pictures, WorkPC –Gnomad demo, pictures demo, –JSDL tool –ls around in the queues –drop a bunch of jobs into a queue –Cd grid/queues, ls around –ExportDirTool from my desktop

30. Genesis II Open Source, OGSA Implementation Genesis II Take-away messages Sufficient body of standards exist with which to build interesting, useful grid systems Available under Apache 2 licenses Very active project! Information and Download Page –http://vcgr.cs.virginia.edu/genesisIIhttp://vcgr.cs.virginia.edu/genesisII Forum –http://www.cs.virginia.edu/forums/viewforum.php?f=26http://www.cs.virginia.edu/forums/viewforum.php?f=26