Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary

2Western Grid Summit Grid Tutorial Acknowledgments Rob Simmonds Mark Fox Phil Rizk Abhishek Gaurav Andrey Mirtchovski Nayden Markatchev

3Western Grid Summit Grid Tutorial Outline Grid Computing Overview Basic Grid Services security, data management, resource management, information services security, data management, resource management, information services Advanced Grid Services meta schedulers, workflow managers, replica location service meta schedulers, workflow managers, replica location service Future of Grid Computing

4Western Grid Summit Grid Tutorial Grid Computing Aimed at providing users with a single system view of a set of distributed resources that span administrative domains Aids in solving large scale problems Aids in collaborative research efforts

5Western Grid Summit Grid Tutorial Current Grid Projects CERN LCG (LHC Computing Grid) GriPhyN (Grid Physics Network) NEES (Network for Earthquake Engineering Simulation) BioSimGrid IVOA (International Virtual Observatory Alliance)

6Western Grid Summit Grid Tutorial WestGrid and Grid Is WestGrid a computational grid? Provides grid enabled resources Security services Security services Data transfer tools Data transfer tools Job submission services Job submission services WestGrid resources can be part of computational grids

7Western Grid Summit Grid Tutorial Global Grid Forum (GGF) Leads the global standardization effort for grid computing Open Grid Services Architecture (OGSA)

8Western Grid Summit Grid Tutorial Open Grid Services Architecture Infrastructure Services Resource Management Services Data Services Context Services Information Services Self-Management Services Security Services Execution Management Services

9Western Grid Summit Grid Tutorial Grid Middleware Globus Toolkit Globus Alliance Globus Alliance Open source Open sourceLegion University of Virginia University of Virginia Now commercial product developed by Avaki Now commercial product developed by Avaki Focuses on data grid / enterprise integration Focuses on data grid / enterprise integration

10Western Grid Summit Grid Tutorial Globus Toolkit Collection of grid services to aid in development of grid applications and environments GT-2 – pre standards (defacto standard) GT-3 – Open Grid Services Infrastructure (OGSI) GT-4 – Web Services Resource Framework (WSRF) – beta version

11Western Grid Summit Grid Tutorial Basic Grid Services Security Services GSI (Grid Security Infrastructure) GSI (Grid Security Infrastructure) Data Management GridFTP GridFTP Resource Management GRAM (Grid Resource Allocation and Management) GRAM (Grid Resource Allocation and Management) Information Services MDS (Monitoring and Discovery Service) MDS (Monitoring and Discovery Service)

12Western Grid Summit Grid Tutorial Grid Security Infrastructure (GSI) Based on Public key cryptography Public key cryptography X.509 certificates X.509 certificates Secure Socket Layer (SSL) Secure Socket Layer (SSL)Provides Mutual authentication Mutual authentication Single sign-on Single sign-on

13Western Grid Summit Grid Tutorial Certificates Used to identify and authenticate users and services Includes four primary pieces of information Subject name Subject name Public key Public key Identity of Certificate Authority (CA) Identity of Certificate Authority (CA) Digital signature of the Certificate Authority Digital signature of the Certificate Authority Provides global name space Encoded in X.509 certificate format (IETF)

14Western Grid Summit Grid Tutorial Mutual Authentication Process of two parties proving to each other who they say they are Both parties must trust CAs that signed each other’s certificates Each party sends certificate to other Make sure certificate is valid Make sure certificate is valid Make sure other party really is the party identified in the certificate Make sure other party really is the party identified in the certificate

15Western Grid Summit Grid Tutorial Single Sign-On Sign-on only once to computational grid Use a proxy to avoid re-entering password Consists of new certificate and private key Consists of new certificate and private key Certificate signed by owner, not CA Certificate signed by owner, not CA Less secure – limited lifetime Less secure – limited lifetime Both proxy certificate and owner’s certificate sent for mutual authentication Both proxy certificate and owner’s certificate sent for mutual authenticationgrid-proxy-init

16Western Grid Summit Grid Tutorial GSI Demo

17Western Grid Summit Grid Tutorial MyProxy Developed by NCSA (National Center for Supercomputing Applications) Credential repository Allows proxy credential to be received from any machine Can allow trusted servers to renew proxy credentials myproxy-init, myproxy-get-delegation

18Western Grid Summit Grid Tutorial MyProxy Demo

19Western Grid Summit Grid Tutorial File Transfers gsiscp scp version based on GSI scp version based on GSI Easy to use, but provides poor performance for large file transfers Easy to use, but provides poor performance for large file transfersGridFTP Based on FTP (File Transfer Protocol) Based on FTP (File Transfer Protocol) GSI security on control and data channels GSI security on control and data channels Modification of TCP buffer sizes Modification of TCP buffer sizes Parallel transfers Parallel transfers Third party transfers Third party transfers

20Western Grid Summit Grid Tutorial File Transfer Performance

21Western Grid Summit Grid Tutorial File Transfer Demo

22Western Grid Summit Grid Tutorial Grid Resource Allocation and Management (GRAM) Provides a single standard interface for remote job submission and resource management Requires users and application developers to learn only one method to gain access to a large variety of local management systems globus-job-run, globus-job-submit

23Western Grid Summit Grid Tutorial Job Submission Demo

24Western Grid Summit Grid Tutorial Monitoring and Discovery Service (MDS) Provides information about availability and status of resources Uses LDAP (Lightweight Directory Access Protocol) to access this information Grid Resource Information Service (GRIS) Provides resource information Provides resource information Grid Index Information Service (GIIS) Provides caching services and search capabilities Provides caching services and search capabilities

25Western Grid Summit Grid Tutorial Advanced Grid Services Meta-schedulersWorkflows Replica Location Service (RLS) Abstract Workflows

26Western Grid Summit Grid Tutorial Meta-Schedulers Condor-G (Condor) University of Wisconsin-Madison University of Wisconsin-Madison Moab Grid Scheduler (Silver) Cluster Resources Inc. Cluster Resources Inc.

27Western Grid Summit Grid Tutorial Condor-G Can submit and manage jobs at multiple sites Uses Globus mechanisms to communicate with resources at different sites Can advertise sites that jobs can be submitted to through Class Ads

28Western Grid Summit Grid Tutorial Class Ad Used to represent characteristics and constraints of resources and jobs Consists of a list of attributes (name/value pairs) Used in matchmaking of jobs to resources

29Western Grid Summit Grid Tutorial Condor-G Demo

30Western Grid Summit Grid Tutorial Workflows A B C1C1 C2C2 D C3C3 E2E2 E1E1

31Western Grid Summit Grid Tutorial DAGMan Directed Acyclic Graph Manager Node = program Edge = dependency Monitors Condor log files to enforce ordering of dependencies

32Western Grid Summit Grid Tutorial DAGMan Input File Programs in the DAG Pre/post processing scripts Dependencies Number of times to retry Macro definitions

33Western Grid Summit Grid Tutorial Sample DAGMan Input File JOB A A.condor JOB B B.condor JOB C C.condor JOB D D.condor PARENT A CHILD B C PARENT B C CHILD D A BC D

34Western Grid Summit Grid Tutorial Rescue DAGs Aids in resubmission of uncompleted portions of workflows Produced if DAGMan can not make any further progress in a DAG due to node failures Same as original DAG but nodes that successfully completed before are marked as DONE

35Western Grid Summit Grid Tutorial Life3D Workflow Life3D Simulation Rendering Animation

36Western Grid Summit Grid Tutorial Life3D Workflow - II WestGrid Grid Research Center blackhole (SFU) lattice (UofC) grc15 octarine Life3D Simulation Rendering Animation 7. Data Storage

37Western Grid Summit Grid Tutorial Life3D Workflow - III Life3d Simulation Rendering Animation Stage Data Stage Data Stage Data Stage Data blackhole lattice grc15 octarine

38Western Grid Summit Grid Tutorial DAGMan Demo

39Western Grid Summit Grid Tutorial Life3D Animation

40Western Grid Summit Grid Tutorial Replica Location Service (RLS) Co-developed by Globus and DataGrid projects Provides a mechanism for registering and locating replicas Maintains mappings between logical names and target names Logical names – unique identifiers for data content that may have one or more physical replicas Logical names – unique identifiers for data content that may have one or more physical replicas Target name – physical location of a replica or other logical names representing the data Target name – physical location of a replica or other logical names representing the data

41Western Grid Summit Grid Tutorial Chimera Virtual Data System Part of the GriPhyN project Distributed with the Virtual Data Toolkit Keeps track of how data products are derived Chimera Pegasus Partial Workflow (VDL) Virtual Data Language Abstract Workflow Concrete Workflow (DAX) (DAG) DAG XML RLS TC MDS (Transformation Catalog)

42Western Grid Summit Grid Tutorial Chimera Input - Virtual Data Language (VDL) Transformation – an executable program Transformation – an executable program Derivation – an execution of the transformation along with logical input and output file names Derivation – an execution of the transformation along with logical input and output file names Output – Abstract Workflow (DAX) Workflow activities independent of specific grid resources Workflow activities independent of specific grid resources

43Western Grid Summit Grid Tutorial Pegasus Input – Abstract Workflow Output – Concrete Workflow (DAG) Workflow activities assigned to specific Gird resources Workflow activities assigned to specific Gird resources Uses RLS to determine data file locations Uses Transformation Catalog to map logical transformations to physical locations Uses MDS to determine available resources

44Western Grid Summit Grid Tutorial Chimera Example add10 add20 infile (blackhole) (lattice) (nexus) outfile (blackhole) transitfile

45Western Grid Summit Grid Tutorial Chimera Demo

46Western Grid Summit Grid Tutorial Summary Security Services (GSI) grid-proxy-init grid-proxy-init myproxy-init myproxy-init myproxy-get-delegation myproxy-get-delegation Data Management (GridFTP) globus-url-copy globus-url-copy Resource Management (GRAM) globus-job-run globus-job-run globus-job-submit globus-job-submit Information Services MDS MDSMeta-scheduler Condor-G Concrete Workflows DAGMan Replica Location Service (RLS) Abstract Workflows Chimera/Pegasus Basic Grid Services Advanced Grid Services

47Western Grid Summit Grid Tutorial Future of Grid Computing Integration of Web Services and Grid Services Web Services Resource Framework (GT-4) Standards are more established Involvement of more developers Involvement of more developers Better tools Better tools Grid-enabled applications Common library interface Common library interface “run_job_on_grid” “run_job_on_grid” Wider spread use

48Western Grid Summit Grid Tutorial Contact Information Cameron Kiddle