1. 3-2.1 Topics Grid Computing Meta-schedulers –Condor-G –Gridway Distributed Resource Management Application (DRMAA) © 2010 B. Wilkinson/Clayton Ferner. Spring 2010 Grid computing course. Modification date: Feb 15, 2010

2. Meta-schedulers Schedule jobs across distributed sites Highly desirable in a Grid computing environment. For a Globus installation, interfaces to local Globus GRAM installation, which in turn interfaces with local job scheduler Uses whatever local scheduler present at each site 3-2.2

3. 3-2.3 Fig 3-17 Meta-scheduler interfacing to Globus GRAM

4. Condor-G A version of Condor that interfaces to Globus environment. Jobs submitted to Condor through Grid universe and directed to Globus job manager (GRAM) 3-2.4 Fig 3-18

5. Simple job description file universe = grid grid_resource = gt4 https://coit-grid03.uncc.edu:8440/ wsrf/services/ManagedJobFactoryService Fork executable = /usr/bin/uptime log = condor_test1.log output = condor_test1.out error = condor_test1.error should_transfer_files = YES when_to_transfer_output = ON_EXIT queue 3-2.5

6. 3-2.6 Communication between user, myProxy server, and Condor-G for long-running jobs Fig 3.19

7. Gridway A meta-scheduler designed specifically for a Grid computing environment Interfaces to Globus components. Project began in 2002. Now open source. Became part of Globus distribution from version 4.0.5 onwards (June 2007). 3-2.7

8. Has ability to match jobs to resources using both static information about job and resources, and dynamic information (resource load) Dynamic scheduling Automatic job migration, including controlled by the job during execution Checking for both fault tolerance and dynamic job migration Reporting and accounting facilities Basic job dependencies (workflow) 3-2.8

9. Can be installed on: Client machines to interact with a distributed system or Server where multiple users access it. Uses file transfer, execution management, and information services of Globus 3-2.9

10. 3-2.10 Globus components used with Gridway Fig 3-20

11. Submitting a Gridway Job Jobs described in a Gridway job template (GWJT). Sample EXECUTABLE = /bin/ls EXECUTABLE = /usr/bin/uptime STDOUT_FILE = stdout.${JOB_ID} STDERR_FILE = stderr.${JOB_ID} If above saved as file myJob.jt, command: gwsubmit -t myJog.jt would cause it to be submitted to be executed on a Grid resource. 3-2.11

12. Job matching Uses REQUIREMENTS and RANK (similar to Condor) REQUIREMENTS expression has to evaluate to TRUE for the execution host to be consider at all for job. RANK expression computed for each host and host with higher ranks used first for job. 3-2.12

13. 3-2.13 Example: RANK = CPU_MHZ

14. Array Jobs Specifying multiple instances of job, with different arguments. EXECUTABLE = myJob.exe ARGUMENTS = ${TASK_ID} STDOUT_FILE = stdout_file.${TASK_ID} STDERR_FILE = stderr_file.${TASK_ID} RANK = CPU_MHZ RANK expression - processors with higher clock frequency preferred (should be coupled with processor type if higher clock frequency to mean higher performance.) Array of 10 instances of myJob.exe could be submitted with: gwsubmit -t myJob.jt -n 10 3-2.14

15. Distributed Resource Management (DRM) systems Term used to cover job schedulers and the like. Several choices of DRM’s for a system, each having different characteristics and modes of operation and different commands and APIs. 3-2.15

16. Distributed Resource Management Application (DRMAA) (pronounced “drama”) Standard set of API’s for submission and control of jobs to DRM’s Bindings in C/C++, Java, Perl, Python, and Ruby for a range of DSMs including (Sun) Grid Engine, Condor, PBS/Torque, LSF and Gridway 3-2.16

17. Scheduler with DRMAA interface 3-2.17 Fig 3.21

18. Example of the use of DRMAA 3-2.18 Fig 3.22

19. Questions 3-2.19