1. Clarens Toolkit Building Blocks for a Simple TeraGrid Gateway Tutorial Conrad Steenberg Julian Bunn, Matthew Graham, Joseph Jacob, Craig Miller, Roy Williams California Institute of Technology TeraGrid 2007 Madison, WI

2. June 2007TeraGrid 20072 Introduction Clarens Toolkit architecture Shell service basics User mapping Job submission Web interface hosting

3. June 2007TeraGrid 20073 Clarens Toolkit Architecture High-performance Apache web server “handler” Written from the ground up in Python since 2001 Multi-protocol communication (XML- RPC/JSON/SOAP) Geared towards scientists – requires very little programming to create web services Supports X509/proxy-based authentication Role-based and Gridmap file based authorization File Upload/download – AJAX hosting See presentation on Tuesday by Julian Bunn

4. June 2007TeraGrid 20074 Clarens Architecture IIClient Web server Web serverService 3 rd party application Clarens Clarens http/https XML-RPC SOAP Java RMI JSON/RPC The Clarens Toolkit Supports: Authentication – User certificates and proxies Access control – to Web Services, remote files Discovery - Web Services and Software Shell – Remote shells with ACL Virtual Organization – membership, management and roles Services – Potentially infinite :-) JavaScript (AJAX), Java, Python, Root (analysis framework)

5. June 2007TeraGrid 20075 Shell Service Basics Shell service can be used to run an executable on the server –Mapping user's credentials to a local system user External API: –id=shell.cmd(“some command”, id) –Each command executed in a sandbox owned by local system user –stdout and stderr files created in sandbox –Sandboxes can be re-used by supplying optional id argument –Files can be uploaded and downloaded to sandbox via web service or HTTP PUT using a form in a browser

6. June 2007TeraGrid 20076 Shell Service Basics II

7. June 2007TeraGrid 20077 User Mapping Map credentials (X509/proxy) to local system user Use classical gridmap-file – "/DC=org/DC=doegrids/OU=People/CN=Conrad Steenberg 178947" conrad Use external callout Site Authorization Service Use Clarens hierarchical VO-based mapping –Hierarchical group structure –Any user or group can be mapped to local user –E.g. “Caltech” group consisting of Roy, Joe, Julian mapped to “caltech” user on remote machine No credential can be mapped to web server user

8. June 2007TeraGrid 20078 Job Submission Client-side framework can be developed using only shell service basics –Needs to be repeated for every access method and every client –Higher level services can be developed using code that make up shell service as a library –Higher level service can then be exposed to outside world E.g. rootlet service Higher level services can provide complex user mapping –E.g. NVO has anonymous, weak, and strong certificates mapping to guest account, nvo account or individual account resp.

9. June 2007TeraGrid 20079 Job Submission II Shell service library use on server: –Write service using tutorial on clarens.sf.net –Import shell service as library: shell_mod=clarens_util.import_module(“shell”) –Obtain desired user mapping: user_name,uid,gid,dn=shell_mod.lookup_user(req) –Execute shell command: shell_mod.exe_worker(req, method_name, cmd, request_id, user_name) –Or wrap user executable with script to submit to PBS/LSF/Condor

10. June 2007TeraGrid 200710 Web Interface Hosting Clarens server handles ordinary web page hosting –Can host html/js files to build AJAX interfaces –Use any popular client-side toolkit (yahoo, google, dojo, etc.) –Clarens provides support for method invocation from javascript –Uses same authentication/authorization for every HTTP GET/PUT request Example: – –jsonrpc("shell.cmd", [“date”,id], date_callback, error_cb)

11. June 2007TeraGrid 200711 Web Interface Hosting II

12. June 2007TeraGrid 200712 Summary Clarens Toolkit allows portal builders a flexible way to build web services –Uses wrist-friendly Python language –Simple design and implementation –Open Source Supports advanced portal requirements –Range of user mapping methods from simple to complex –Powerful access control mechanisms –Web interface hosting Examples of use include –NESSSI services –rootlets in HEP

13. June 2007TeraGrid 200713 Contact Info Clarens: http://clarens.sf.nethttp://clarens.sf.net –Complete manual –One-liner installation for x86-Linux –Service tutorial –Bug-tracking/mailing lists NVO: http://www.us-vo.org/nesssi/index.cfmhttp://www.us-vo.org/nesssi/index.cfm –Example services

14. June 2007TeraGrid 200714

15. June 2007TeraGrid 200715

16. June 2007TeraGrid 200716

17. June 2007TeraGrid 200717