1. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 1 GridX1 Services Project A. Agarwal, A. Berman, A. Charbonneau, R. Desmarais, I. Gable, R. Impey, D. Leske, G. Mateescu, A. Norton, W. Podaima, S. Popov, D. Quesnel S. Ramage, S. Reynen, R. Sobie, C. Usher, D. Vanderster CANARIE Inc., Ottawa, Ontario, Canada National Research Council, Ottawa, Ontario, Canada University of Victoria, Victoria, British Columbia, Canada

2. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 2 Overview Motivation Concepts –Web Services –Grid Services –Second Generation Web Services Project Goals –Determine Requirements for a Grid –Evaluate web services based grid computing –Implement a web services based grid –Provide users with an easy to use interface to a web services grid Requirements –Meta scheduler and Registry Requirements –Monitoring Requirements –Grid Projects Evaluation Implementation –GridX1 Services Project Test bed and Middleware –Meta Scheduler –Registry –Monitoring –Job Submission Tool Summary

3. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 3 Motivation The increased use of grid technology for scientific analysis has raised several issues over the passed five to six years. Users of the grid have to be quite knowledgeable to implement client tools that use grid resources. Creating extensions to add functionality to a grid are very difficult. Administration of grid users and resources are time consuming. To ease these burdens, research in providing web services based standards to manage grid communication is being developed to ease development, scalability, accessibility and usability.

4. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 4 Concepts :web services A web service is any program which makes itself available over the internet and uses XML as a way of communicating with other programs. [1] Web services can be put together to form an architecture. For example, the Service Oriented Architecture. The Service Oriented Architecture is a common way of providing users access to a service. These traditional web services are considered stateless in that they have no memory of passed events. [2] Service Oriented Architecture [2] Basic web service usage [4]

5. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 5 Concepts : grid services Grid Services, such as those provided by Globus 2, provide the user with the ability to access large amounts of processor power and data storage. The Job Manager service is the main service provided to users. In particular globus offers a gram service which accepts user job submission. Globus Grid Services [3]

6. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 6 Concepts :Second Generation Web Services Second Generation Web Services, such as Globus 4, is an extension of web services. In particular the OGSA is an architecture that can be used to implement second generation web services [4]. The WSRF (Web Services Resource Framework) implemented by Globus and specifies Stateful web services as required by the OGSA. The WSRF uses web services and gives them a memory. In particular the WSRF describes a Web Service resource. A Web Service Resource is a web service with a resource (such as memory). This leads to stateful web services which is required by a web services grid. Globus services offered using WSRF [4]

7. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 7 Project Goals :Determine Requirements for a Grid Typically a grid consists of a Meta scheduler, Registry, Monitor and one or more grid resources. Before we go ahead and construct a web services based grid, it would be advantageous to understand what is required of the grid. We must know: Meta scheduler requirements Registry requirements Monitoring requirements Typical Grid Components

8. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 8 Project Goals :Evaluate web services based grid computing To gain insight into how grids are currently being implemented, an evaluation of several popular grids (including gridX1) was completed and the grids were assessed by a set of grid requirements which were elicited from many knowledgeable grid guru’s. Several organizations were chosen to evaluate their grid services implementation, these include GridX1, Teragrid, Gridbus, EGEE, NorduGrid and the OSG

9. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 9 Project Goals :Implement a web services based grid Once the evaluation of current grids is completed, a web services grid will be implemented on it. This will require: Creation of a test bed : setup networked machines to be assigned as resources or registry or meta scheduler or monitor ….. Installation of selected web services grid middleware Installation/Creation of a Meta scheduler Installation/Creation of a Registry Installation/Creation of a Monitor Setup of Selected Sites to use the test bed

10. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 10 Project Goals :Provide users with an easy to use interface to a web services grid To provide a better experience when using the web services based grid, a user interface to use the grid will be provided. This could be in the form of a client tool or possibly with a web browser. Client Tool [5] Web Broswer [6]

11. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 11 Requirements :Meta Scheduler Requirements Meta Scheduler requirements document http://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf http://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf Primary Meta Scheduling Requirements Use web services to receive jobs from clients Use web services to schedule jobs to resources ***put picture here to clarify usage****

12. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 12 Requirements :Registry Requirements Registry Requirements document http://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf http://wiki.gridx1.ca/twiki/svndocs/Software RequirementsSpecification.pdf Primary Registry Requirements Use web services to receive resource updates Allow web services access to registry information ***put picture here to clarify****

13. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 13 Requirements :Monitoring Requirements Monitoring Requirements document http://wiki.gridx1.ca/twiki/pub/Main/SvnDo cs/GridMonitoringRequirementsAndDe sign.pdf Primary Monitoring Requirements Allow web browser access to stored monitored information Use Web services to determine status of grid to be stored in local database ****insert picture here to clarify***

14. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 14 Requirements :Grid Projects Evaluation One of the project Goals was to complete an evaluation of current grid projects. This evaluation would be used to determine how best to implement a web services grid of our own. The requirements document can be found http://wiki.gridx1.ca/twiki/svndocs/SoftwareRequire mentsSpecification.pdf http://wiki.gridx1.ca/twiki/svndocs/SoftwareRequire mentsSpecification.pdf The analysis document can be found http://wiki.gridx1.ca/twiki/pub/Main/SvnDocs/Regis tryAndMetaschedulerReview.pdf http://wiki.gridx1.ca/twiki/pub/Main/SvnDocs/Regis tryAndMetaschedulerReview.pdf From the evaluation, we concluded that Teragrid, Gridbus and EGEE grid implementations would be the best grids to follow for the implementation of our own web services based grid.

15. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 15 Implementation : GridX1 Services Project Test bed and Middleware Having Chosen Teragrid as a model to implement our own web services grid, we need computers to install and test grid software. To do this, we created a gridX1 services project test bed. Currently the test bed consists of 15 machines. This includes: –2 Registries –3 Meta schedulers –4 Head Nodes –4 test worker nodes –2 monitors GridX1 Services Project Test bed Document: http://wiki.gridx1.ca/twiki/svndocs/TestBed.pdf http://wiki.gridx1.ca/twiki/svndocs/TestBed.pdf To facilitate web services communication between the machines, Globus 4 was chosen.

16. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 16 Implementation :Meta Scheduler The Meta Scheduler is the most important component of a grid along with grid middleware to handle the web services communication. Globus 4 was chosen to be implemented on the test bed grid. Globus 4 is web services bases so it meets are need to use web services as a communication standard. Globus has a condor job manager script as well which Gabriel Mateescu has been re-writing to serve our purposes. This script can be used to turn Globus submitted jobs into condor jobs which can then be scheduled. The Meta Scheduler chosen to be implemented is CondorG. CondorG is Globus aware and can use Globus to re-submit jobs to other Globus resources. Condor Meta scheduler using Globus 4 web services

17. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 17 Implementation :Registry The Registry is the next most important component of a grid. In this case, Globus 4 comes with a built in registry service called an index service. The index service however, did not completely meet are needs and we had to add two programs created by Andre Charbonneau to make it useful to us. Along with the Globus index server, we added a resource extractor for each resource head node wishing to publish to our registry, and a resource class ad extractor which is needed by the meta scheduler to get information on registered resources. Registry Index service using web services

18. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 18 Implementation :Monitoring The monitor is used to detect failures in the grid. An example of a failure would be the meta scheduler machine shutting down. The testbed monitor can currently be viewed at www.ugdev04.phys.uvic.ca:8080/G lobusTestbedMonitor/html/index.ht ml www.ugdev04.phys.uvic.ca:8080/G lobusTestbedMonitor/html/index.ht ml There are as well some webservices available to execute jobs from a web page as well. Currently being developed by Ron Desmarais and Sergey Popov

19. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 19 Implementation : Job Submission Tool The job submission tool written by Andre Charbonneau A user can use this tool to submit jobs to the testbed. Andre will demonstrate this tool next.

20. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 20 Summary After an determine requirements of a grid system, a evaluation of several grid projects measured against the requirements was completed. A test bed was put together to evaluate globus 4 web services middle ware with the condor meta scheduler. A working prototype is currently implemented and work on user submission interfacing, monitoring and meta scheduling continues. We had several coops over this past year that helped greatly with what we have so far but are no longer with us. So I would like to thank. Chris Usher, other guy, Simon Ramage, Patrick …… for their work!

21. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 21 References [1] web services definition, http://www.google.ca/url?sa=X&start=1&oi=define&q=http://www.inter networld365.com/Content/Pages/Glossary.aspx http://www.google.ca/url?sa=X&start=1&oi=define&q=http://www.inter networld365.com/Content/Pages/Glossary.aspx [2] services oriented architecture, www7b.software.ibm.com [3] grid services image, http://www.kato.mvc.mcc.ac.uk/blog/NGS/files/NGS/images/Globus- Config.gif http://www.kato.mvc.mcc.ac.uk/blog/NGS/files/NGS/images/Globus- Config.gif [4] Globus Programers Tutorial, http://gdp.globus.org/gt4- tutorial/download/progtutorial-pdf_0.2.1.tar.gzhttp://gdp.globus.org/gt4- tutorial/download/progtutorial-pdf_0.2.1.tar.gz [5] Client Interface, http://www.csse.monash.edu.au/~azaslavs/DAME/Client%20Interface.jpg http://www.csse.monash.edu.au/~azaslavs/DAME/Client%20Interface.jpg

22. Daniel Vanderster University of Victoria National Research Council and the University of Victoria 22 References :continued [6] web interface, http://www.openshark.com/products/LF/webclient.GIF http://www.openshark.com/products/LF/webclient.GIF