1. Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary

2. 2Western Grid Summit - 2005Grid Tutorial Acknowledgments Rob Simmonds Mark Fox Phil Rizk Abhishek Gaurav Andrey Mirtchovski Nayden Markatchev

3. 3Western Grid Summit - 2005Grid Tutorial Outline Grid Computing Overview Basic Grid Services security, data management, resource management, information services security, data management, resource management, information services Advanced Grid Services meta schedulers, workflow managers, replica location service meta schedulers, workflow managers, replica location service Future of Grid Computing

4. 4Western Grid Summit - 2005Grid Tutorial Grid Computing Aimed at providing users with a single system view of a set of distributed resources that span administrative domains Aids in solving large scale problems Aids in collaborative research efforts

5. 5Western Grid Summit - 2005Grid Tutorial Current Grid Projects CERN LCG (LHC Computing Grid) GriPhyN (Grid Physics Network) NEES (Network for Earthquake Engineering Simulation) BioSimGrid IVOA (International Virtual Observatory Alliance)

6. 6Western Grid Summit - 2005Grid Tutorial WestGrid and Grid Is WestGrid a computational grid? Provides grid enabled resources Security services Security services Data transfer tools Data transfer tools Job submission services Job submission services WestGrid resources can be part of computational grids

7. 7Western Grid Summit - 2005Grid Tutorial Global Grid Forum (GGF) Leads the global standardization effort for grid computing Open Grid Services Architecture (OGSA)

8. 8Western Grid Summit - 2005Grid Tutorial Open Grid Services Architecture Infrastructure Services Resource Management Services Data Services Context Services Information Services Self-Management Services Security Services Execution Management Services

9. 9Western Grid Summit - 2005Grid Tutorial Grid Middleware Globus Toolkit Globus Alliance Globus Alliance Open source Open sourceLegion University of Virginia University of Virginia Now commercial product developed by Avaki Now commercial product developed by Avaki Focuses on data grid / enterprise integration Focuses on data grid / enterprise integration

10. 10Western Grid Summit - 2005Grid Tutorial Globus Toolkit Collection of grid services to aid in development of grid applications and environments GT-2 – pre standards (defacto standard) GT-3 – Open Grid Services Infrastructure (OGSI) GT-4 – Web Services Resource Framework (WSRF) – beta version

11. 11Western Grid Summit - 2005Grid Tutorial Basic Grid Services Security Services GSI (Grid Security Infrastructure) GSI (Grid Security Infrastructure) Data Management GridFTP GridFTP Resource Management GRAM (Grid Resource Allocation and Management) GRAM (Grid Resource Allocation and Management) Information Services MDS (Monitoring and Discovery Service) MDS (Monitoring and Discovery Service)

12. 12Western Grid Summit - 2005Grid Tutorial Grid Security Infrastructure (GSI) Based on Public key cryptography Public key cryptography X.509 certificates X.509 certificates Secure Socket Layer (SSL) Secure Socket Layer (SSL)Provides Mutual authentication Mutual authentication Single sign-on Single sign-on

13. 13Western Grid Summit - 2005Grid Tutorial Certificates Used to identify and authenticate users and services Includes four primary pieces of information Subject name Subject name Public key Public key Identity of Certificate Authority (CA) Identity of Certificate Authority (CA) Digital signature of the Certificate Authority Digital signature of the Certificate Authority Provides global name space Encoded in X.509 certificate format (IETF)

14. 14Western Grid Summit - 2005Grid Tutorial Mutual Authentication Process of two parties proving to each other who they say they are Both parties must trust CAs that signed each other’s certificates Each party sends certificate to other Make sure certificate is valid Make sure certificate is valid Make sure other party really is the party identified in the certificate Make sure other party really is the party identified in the certificate

15. 15Western Grid Summit - 2005Grid Tutorial Single Sign-On Sign-on only once to computational grid Use a proxy to avoid re-entering password Consists of new certificate and private key Consists of new certificate and private key Certificate signed by owner, not CA Certificate signed by owner, not CA Less secure – limited lifetime Less secure – limited lifetime Both proxy certificate and owner’s certificate sent for mutual authentication Both proxy certificate and owner’s certificate sent for mutual authenticationgrid-proxy-init

16. 16Western Grid Summit - 2005Grid Tutorial GSI Demo

17. 17Western Grid Summit - 2005Grid Tutorial MyProxy Developed by NCSA (National Center for Supercomputing Applications) Credential repository Allows proxy credential to be received from any machine Can allow trusted servers to renew proxy credentials myproxy-init, myproxy-get-delegation

18. 18Western Grid Summit - 2005Grid Tutorial MyProxy Demo

19. 19Western Grid Summit - 2005Grid Tutorial File Transfers gsiscp scp version based on GSI scp version based on GSI Easy to use, but provides poor performance for large file transfers Easy to use, but provides poor performance for large file transfersGridFTP Based on FTP (File Transfer Protocol) Based on FTP (File Transfer Protocol) GSI security on control and data channels GSI security on control and data channels Modification of TCP buffer sizes Modification of TCP buffer sizes Parallel transfers Parallel transfers Third party transfers Third party transfers

20. 20Western Grid Summit - 2005Grid Tutorial File Transfer Performance

21. 21Western Grid Summit - 2005Grid Tutorial File Transfer Demo

22. 22Western Grid Summit - 2005Grid Tutorial Grid Resource Allocation and Management (GRAM) Provides a single standard interface for remote job submission and resource management Requires users and application developers to learn only one method to gain access to a large variety of local management systems globus-job-run, globus-job-submit

23. 23Western Grid Summit - 2005Grid Tutorial Job Submission Demo

24. 24Western Grid Summit - 2005Grid Tutorial Monitoring and Discovery Service (MDS) Provides information about availability and status of resources Uses LDAP (Lightweight Directory Access Protocol) to access this information Grid Resource Information Service (GRIS) Provides resource information Provides resource information Grid Index Information Service (GIIS) Provides caching services and search capabilities Provides caching services and search capabilities

25. 25Western Grid Summit - 2005Grid Tutorial Advanced Grid Services Meta-schedulersWorkflows Replica Location Service (RLS) Abstract Workflows

26. 26Western Grid Summit - 2005Grid Tutorial Meta-Schedulers Condor-G (Condor) University of Wisconsin-Madison University of Wisconsin-Madison Moab Grid Scheduler (Silver) Cluster Resources Inc. Cluster Resources Inc.

27. 27Western Grid Summit - 2005Grid Tutorial Condor-G Can submit and manage jobs at multiple sites Uses Globus mechanisms to communicate with resources at different sites Can advertise sites that jobs can be submitted to through Class Ads

28. 28Western Grid Summit - 2005Grid Tutorial Class Ad Used to represent characteristics and constraints of resources and jobs Consists of a list of attributes (name/value pairs) Used in matchmaking of jobs to resources

29. 29Western Grid Summit - 2005Grid Tutorial Condor-G Demo

30. 30Western Grid Summit - 2005Grid Tutorial Workflows A B C1C1 C2C2 D C3C3 E2E2 E1E1

31. 31Western Grid Summit - 2005Grid Tutorial DAGMan Directed Acyclic Graph Manager Node = program Edge = dependency Monitors Condor log files to enforce ordering of dependencies

32. 32Western Grid Summit - 2005Grid Tutorial DAGMan Input File Programs in the DAG Pre/post processing scripts Dependencies Number of times to retry Macro definitions

33. 33Western Grid Summit - 2005Grid Tutorial Sample DAGMan Input File JOB A A.condor JOB B B.condor JOB C C.condor JOB D D.condor PARENT A CHILD B C PARENT B C CHILD D A BC D

34. 34Western Grid Summit - 2005Grid Tutorial Rescue DAGs Aids in resubmission of uncompleted portions of workflows Produced if DAGMan can not make any further progress in a DAG due to node failures Same as original DAG but nodes that successfully completed before are marked as DONE

35. 35Western Grid Summit - 2005Grid Tutorial Life3D Workflow Life3D Simulation Rendering Animation

36. 36Western Grid Summit - 2005Grid Tutorial Life3D Workflow - II WestGrid Grid Research Center blackhole (SFU) lattice (UofC) grc15 octarine 1. 2. Life3D Simulation 3. 4. Rendering 5. 6. Animation 7. Data Storage

37. 37Western Grid Summit - 2005Grid Tutorial Life3D Workflow - III Life3d Simulation Rendering Animation Stage Data Stage Data Stage Data Stage Data blackhole lattice grc15 octarine

38. 38Western Grid Summit - 2005Grid Tutorial DAGMan Demo

39. 39Western Grid Summit - 2005Grid Tutorial Life3D Animation

40. 40Western Grid Summit - 2005Grid Tutorial Replica Location Service (RLS) Co-developed by Globus and DataGrid projects Provides a mechanism for registering and locating replicas Maintains mappings between logical names and target names Logical names – unique identifiers for data content that may have one or more physical replicas Logical names – unique identifiers for data content that may have one or more physical replicas Target name – physical location of a replica or other logical names representing the data Target name – physical location of a replica or other logical names representing the data

41. 41Western Grid Summit - 2005Grid Tutorial Chimera Virtual Data System Part of the GriPhyN project Distributed with the Virtual Data Toolkit Keeps track of how data products are derived Chimera Pegasus Partial Workflow (VDL) Virtual Data Language Abstract Workflow Concrete Workflow (DAX) (DAG) DAG XML RLS TC MDS (Transformation Catalog)

42. 42Western Grid Summit - 2005Grid Tutorial Chimera Input - Virtual Data Language (VDL) Transformation – an executable program Transformation – an executable program Derivation – an execution of the transformation along with logical input and output file names Derivation – an execution of the transformation along with logical input and output file names Output – Abstract Workflow (DAX) Workflow activities independent of specific grid resources Workflow activities independent of specific grid resources

43. 43Western Grid Summit - 2005Grid Tutorial Pegasus Input – Abstract Workflow Output – Concrete Workflow (DAG) Workflow activities assigned to specific Gird resources Workflow activities assigned to specific Gird resources Uses RLS to determine data file locations Uses Transformation Catalog to map logical transformations to physical locations Uses MDS to determine available resources

44. 44Western Grid Summit - 2005Grid Tutorial Chimera Example add10 add20 infile (blackhole) (lattice) (nexus) outfile (blackhole) transitfile

45. 45Western Grid Summit - 2005Grid Tutorial Chimera Demo

46. 46Western Grid Summit - 2005Grid Tutorial Summary Security Services (GSI) grid-proxy-init grid-proxy-init myproxy-init myproxy-init myproxy-get-delegation myproxy-get-delegation Data Management (GridFTP) globus-url-copy globus-url-copy Resource Management (GRAM) globus-job-run globus-job-run globus-job-submit globus-job-submit Information Services MDS MDSMeta-scheduler Condor-G Concrete Workflows DAGMan Replica Location Service (RLS) Abstract Workflows Chimera/Pegasus Basic Grid Services Advanced Grid Services

47. 47Western Grid Summit - 2005Grid Tutorial Future of Grid Computing Integration of Web Services and Grid Services Web Services Resource Framework (GT-4) Standards are more established Involvement of more developers Involvement of more developers Better tools Better tools Grid-enabled applications Common library interface Common library interface “run_job_on_grid” “run_job_on_grid” Wider spread use

48. 48Western Grid Summit - 2005Grid Tutorial Contact Information Cameron Kiddle kiddlec@cpsc.ucalgary.ca http://grid.ucalgary.ca/ http://pages.cpsc.ucalgary.ca/~kiddlec/