Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003

Progress Project Duration: December 2001 – May 2003 Deployment: June 2003-December 2003 Project Partners SUN Microsystems Poland PSNC IBCh Poznań Cyfronet AMM, Kraków Technical University Łódź Co-funded by The State Committee for Scientific Research (KBN) and SUN Microsystems Poland

PROGRESS Goals Provide computational portal for scientists Provide easy to use user interface Provide Grid middleware infrastructure Make use of distributed cluster of SUN SF6800 is a pilot testbed installation Make use of PIONIER network infrastructure

PROGRESS Testbed Cluster of 80 processors Networked Storage of 1,3 TB Software: ORACLE, HPC Cluster Tools, Sun ONE, Sun Grid Engine

Architecture - overview PORTAL HPC RESOURCES GRID MANAGEMENT SYSTEM GRID SERVICE PROVIDER 4-tier new grid-portal environment PORTAL HPC RESOURCES GRID MANAGEMENT SYSTEM 3-tier classical grid-portal environment

Grid Broker - goals Provide universal mechanism of resource management for Grid environment Make use of common lower level Grid infrastructure (Globus) Service oriented approach Well-defined universal interface, implemented in common technology (Web Services) Modular architecture

Grid Broker - functionality Ability to choose the best resource for the job execution, according to Job Description and chosen mapping algorithm; Ability to submit the GRMS Task according to provided Job Description; Ability to submit a set of tasks with precedence constraints (sequence/parallel structure) Provides information about the Task status; Provides other information about Tasks (name of host where the Task is/was running, start time, finish time);

Grid Broker – functionality (2) Provides list of resources that fullfills requirements provided in Job Description); Provides a list of Tasks submitted by given user; Ability to transfer input and output files (GridFTP, GASS, Data Management System); Ability to cancel the Task; Provides notification mechanism

Grid Broker - overview Broker Progress Grid Middleware Data Management System MDS GRAM GridFTP GASS Grid Infrastructure PROGRESS Portal User Access Layer Information Service

Grid Broker - details Resource Discovery Broker Task Manager Web Service Interface PROGRESS Portal System Layer Services PROGRESS Services Workflow Mngmt. Task Registry DB

Grid Broker - modules Web Service Interface Provides GSI enabled web service interface for Clients (PROGRESS Portal) Supports callback communication to Client (notification service) Workflow Management Module Creates workflow graph of tasks from Job Description Puts tasks to database Controls of tasks execution according to precedence constraints

Grid Broker – modules (2) Broker Module Manages process of task submittion Chooses the best resources for task execution Transfers executable to chosen resource Transfers input and output files Resource Discovery Module Finds resources that fullfills requirements described in Job Description Provides information about resources, required for task scheduling

Grid Broker – modules (3) Task Manager Module Ability to check current status of task Ability to cancel running task Monitors for status changes of running task Task Registry Stores information about Task execution (start time, finish time, machine where executed, current status, Job Description)

Job Description - XRSL Provides ability to express precedence constraints of Task Grid Broker’s Job contains one or more Tasks Tasks are grouped into sequence or parallel execution

Job Description - schema

Parallel/sequence structure Job1 single Task1 Job3 parallel Job2 sequence Task2Task3 Task1 Task2 Task3 Task2aTask2bTask3c Task2a Task2b Task2c

Job Description - Task Task executable file location arguments file argument (files which have to be present in working directory of running executable) environment variables standard input standard output standard error

Job Description - Task Resource requirements name of host for job execution operating system required local resource management system minimum memory required minimum number of cpus required minimum speed of cpu other parameter passed directly to Globus GRAM

XRSL - example 4 mydata.dat 5

XRSL – example (2) 4 4

Thank you