1. © 2007 UC Regents1 Track 1: Cluster and Grid Computing NBCR Summer Institute Session 1.1: Introduction to Cluster and Grid Computing July 31, 2007 Wilfred Li, wilfred@sdsc.eduwilfred@sdsc.edu Zhaohui Ding, zhding@ucsd.eduzhding@ucsd.edu Based on Slides from Nadya Williams

2. © 2007 UC Regents2 HPC Cluster Architecture Frontend Node Public Ethernet Private Ethernet Network Application Network (Optional) Node Power Distribution (Net addressable units as option)

3. © 2007 UC Regents3 Grids

4. © 2007 UC Regents4 What is the Grid?  1998: “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high- end computational capabilities” - Carl Kesselman and Ian Foster  Grid computing is an emerging computing model that provides the ability to perform higher throughput computing by taking advantage of many networked computers to model a virtual computer architecture that is able to distribute process execution across a parallel infrastructure - from WikipediA

5. © 2007 UC Regents5 Grid’s key elements  Coordination of resources that are subject to decentralized control  Resources from different domains (VO, company, department)  Users from different domains  Use of standard, open general-purpose protocols and interfaces  Authentication/authorization  Resource discovery/access  Delivers non-trivial quality of service  Utility of combined system >> sum of parts From “What is the grid? A three point checklist” - Ian Foster, 2002

6. © 2007 UC Regents6 HPC clusters vs. Grids 1. Closely coupled, operates as a single computing resource 2. Often has high-performance networking interconnects 3. Uses specialized OS designed to appear as a single computing resource 4. SIMD/MIMD models of work execution 1. Heterogeneous, operates as a generalized computing resource 2. Can use high-performance or standard interconnects 3. Individual systems are not specialized; based on standard machines and OSes 4. HPC execution + different operations or instructions. 5. Grid combines – monitoring of the nodes – queuing system for work units

7. © 2007 UC Regents7 Light and dark sides of Grid  Gives substantial computing power for simulations, complex computations and analisys  Existing resources may be better exploited  Through a VO can provide redundancy and share resources  May offer scalability via incremental increase of resources  Indeterminate quality of service  Commercial grid services are $$$  Rapidly changing infrastructure  Grid technologies are VARIED, some are IMMATURE  Some require additional specialized high-capacity communication links

8. © 2007 UC Regents8 How Rocks handles complexities continued  Extend cluster with rolls  Rolls are containers for software packages and their configuration scripts  Rolls dissect a monolithic distribution  100% automated nodes installation & configuration  from distribution assembled at the system installation time  Node is always in a known state Down Installing Running

9. © 2007 UC Regents9 Grid Roll

10. © 2007 UC Regents10 Globus Commands - setup  Generate a certificate request file  grid-cert-request type name and passphrase when requested $HOME/.globus/ generated with 3 files: usercert.pem (empty) usercert_request.pem userkey.pem  Start SSL certificate proxy (similar to ssh-agent)  grid-proxy-init type passphrase once for lifetime on the proxy  Get proxy information  grid-proxy-info  Authenticate to a remote gatekeeper  globusrun -a -r host.name.here

11. © 2007 UC Regents11 Globus Commands  globusrun - basic command to submit a job where attributes are specified in RSL There are several ways to specify RSL: 1. As a file globusrun -o -r -f rsl-file 2. Resource option plus command line argument globusrun -r rocks-32.sdsc.edu -o '&(executable=/bin/env)’ 3. Command line arguments globusrun -o "+( &(resourceManagerContact=rocks-32.sdsc.edu) (executable=/bin/env))" -o is used to capture stdout and stderr to terminal -r is used to specify resource contact manager & (jobtype = single) (executable = "/bin/uname") (arguments = "-a") (count = 2) (stdout = "uname.out" )

12. © 2007 UC Regents12 Globus Commands  globus-job-run - higher level job submission. Takes the command line arguments, generates the rsl file and invokes globusrun. 1. Submit a simple job globus-job-run rocks-32.sdsc.edu /bin/hostname 2. Submit job with staging of executable  Create a shell script testrun.sh  globus-job-run rocks-32.sdsc.edu -s testrun.sh 3. Specify all options in a file  globus-job-run -file myfile 4. Use file copy then submit job without staging  globus-job-run rocks-32.sdsc.edu /bin/mkdir dir1  globus-url-copy file://$HOME/testrun \ gsiftp://rocks-32.sdsc.edu/home/nadya/dir1/testrun.sh  globus-job-run rocks-32.sdsc.edu /bin/chmod +x $HOME/dir1/testrun.sh  globus-job-run rocks-32.sdsc.edu $HOME/dir1/testrun.sh #/bin/bash date hostname -f uptime rocks32.sdsc.edu /bin/env

13. © 2007 UC Regents13 How to use the grid without knowing about the grid

14. © 2007 UC Regents14 SOA based Pervasive Computing: How to shop for the best deal?

15. © 2007 UC Regents15 How to deploy applications easily to the grid environment? Opal – a generic web service wrapper – http://nbcr.net/services/ http://nbcr.net/services/

16. © 2007 UC Regents16 How to leverage the WSRF functionalities? Opal Service Provider

17. © 2007 UC Regents17 How to handle the user certificate creation? Grid Account Management Architecture (GAMA)

18. © 2007 UC Regents18 Opal Web Service based Workflows

19. © 2007 UC Regents19 Opal Based Web Service in the Python Molecular Viewer